2024-04-25T11:06:05.112 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T11:06:05.112 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} --verbose --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123 --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/orig.config.yaml 2024-04-25T11:06:05.162 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T11:06:05.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673123' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - Replacing daemon mds - SLOW_OPS - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhMlJn51XAwCMqALUCDGHwKhofHRhHNVUyws1BBBbvddkzmyoJ8pvvTWO0QJINIKuVttVKdlB1vfVTsFPRx53w= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4+u+4Hm0cZbDUoP3PAPqtfxjIWHTQkiFhAIU6LidCBYLCwNpcF8ElO7HNne+NYRSUXDP1giHO1/eP5PU4RKoA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T11:06:05.255 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T11:06:05.256 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T11:06:05.256 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T11:06:05.258 INFO:teuthology.task.internal:Checking packages... 2024-04-25T11:06:05.281 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T11:06:05.281 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T11:06:05.281 INFO:teuthology.packaging:ref: None 2024-04-25T11:06:05.281 INFO:teuthology.packaging:tag: None 2024-04-25T11:06:05.281 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:06:05.281 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:06:05.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:06:05.498 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:06:05.499 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T11:06:05.506 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T11:06:05.506 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T11:06:05.521 INFO:teuthology.task.internal:Saving configuration 2024-04-25T11:06:05.539 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T11:06:05.544 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T11:06:05.569 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 10:59:52.998688', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGthrxZ7A7443WPvPS/v5XZXnmZsNJCk6tU9scELe36D'} 2024-04-25T11:06:05.592 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 10:59:53.000681', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFp8TMmmID5H7sETphuvugbj+GUQkyny4zO5YywEouxK'} 2024-04-25T11:06:05.592 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T11:06:05.597 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-25T11:06:05.597 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-25T11:06:05.597 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T11:06:05.666 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f036ead93a0>, signals=[15]) 2024-04-25T11:06:05.666 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T11:06:05.690 INFO:teuthology.task.internal:Opening connections... 2024-04-25T11:06:05.690 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:06:05.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:06:05.770 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:06:05.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:06:05.849 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T11:06:05.854 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-04-25T11:06:05.859 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-04-25T11:06:05.859 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-25T11:06:05.909 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T11:06:05.909 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2024-04-25T11:06:05.909 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="22.04" 2024-04-25T11:06:05.909 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T11:06:05.909 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=jammy 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T11:06:05.910 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=jammy 2024-04-25T11:06:05.911 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-25T11:06:05.934 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-04-25T11:06:05.939 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-04-25T11:06:05.939 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Ubuntu" 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="22.04" 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_CODENAME=jammy 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:ID=ubuntu 2024-04-25T11:06:05.986 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE=debian 2024-04-25T11:06:05.987 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T11:06:05.987 INFO:teuthology.orchestra.run.smithi157.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T11:06:05.987 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T11:06:05.987 INFO:teuthology.orchestra.run.smithi157.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T11:06:05.987 INFO:teuthology.orchestra.run.smithi157.stdout:UBUNTU_CODENAME=jammy 2024-04-25T11:06:05.987 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-04-25T11:06:06.017 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T11:06:06.028 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T11:06:06.032 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T11:06:06.033 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-04-25T11:06:06.035 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-04-25T11:06:06.039 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T11:06:06.045 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T11:06:06.045 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-04-25T11:06:06.080 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-04-25T11:06:06.121 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T11:06:06.177 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T11:06:06.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T11:06:06.191 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T11:06:06.191 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T11:06:06.192 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T11:06:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:06:06.196 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-04-25T11:06:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:06:06.197 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-04-25T11:06:06.242 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-56-generic 2024-04-25T11:06:06.243 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.15.0-56-generic 2024-04-25T11:06:06.243 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-04-25T11:06:06.244 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-56-generic 2024-04-25T11:06:06.244 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.15.0-56-generic 2024-04-25T11:06:06.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-04-25T11:06:06.377 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-04-25T11:06:06.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-04-25T11:06:06.532 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:06:06.554 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T11:06:06.629 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T11:06:06.661 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:06:06.664 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T11:06:06.673 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T11:06:06.721 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T11:06:06.760 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T11:06:06.841 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T11:06:06.881 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T11:06:06.885 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T11:06:06.885 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T11:06:06.902 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T11:06:06.907 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T11:06:06.909 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T11:06:06.909 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T11:06:06.914 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T11:06:06.922 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T11:06:06.924 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T11:06:06.926 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T11:06:06.959 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T11:06:06.959 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T11:06:06.960 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T11:06:06.960 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T11:06:06.961 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T11:06:06.961 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T11:06:06.962 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T11:06:06.962 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T11:06:06.962 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T11:06:06.980 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T11:06:07.000 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T11:06:07.083 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T11:06:07.101 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T11:06:07.105 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T11:06:07.221 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T11:06:07.395 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T11:06:07.517 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T11:06:07.532 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T11:06:07.534 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T11:06:07.544 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T11:06:07.544 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T11:06:07.545 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T11:06:07.564 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T11:06:07.571 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T11:06:07.576 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T11:06:07.577 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T11:06:07.579 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T11:06:07.586 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T11:06:07.587 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T11:06:07.588 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T11:06:07.639 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T11:06:07.650 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T11:06:07.650 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T11:06:07.651 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T11:06:07.678 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T11:06:07.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T11:06:07.679 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T11:06:07.685 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T11:06:07.685 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T11:06:07.696 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T11:06:07.844 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T11:06:07.855 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T11:06:07.876 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T11:06:08.262 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T11:06:08.267 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T11:06:08.274 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T11:06:08.280 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T11:06:08.282 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T11:06:08.282 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T11:06:08.285 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T11:06:08.294 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T11:06:08.295 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T11:06:08.297 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T11:06:08.481 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T11:06:08.518 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T11:06:08.532 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T11:06:08.535 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T11:06:08.538 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T11:06:08.541 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T11:06:08.546 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T11:06:08.549 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T11:06:08.566 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T11:06:08.568 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T11:06:08.733 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T11:06:08.817 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T11:06:08.934 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T11:06:08.995 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T11:06:09.025 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T11:06:12.492 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 13.3 MB in 2s (7,244 kB/s) 2024-04-25T11:06:13.300 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 13.3 MB in 3s (5,170 kB/s) 2024-04-25T11:06:13.842 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:06:13.861 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T11:06:13.920 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:06:14.122 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-04-25T11:06:14.123 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-04-25T11:06:14.300 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:06:14.300 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:06:14.300 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:06:14.300 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:06:14.301 INFO:teuthology.orchestra.run.smithi157.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T11:06:14.301 INFO:teuthology.orchestra.run.smithi157.stdout: python2.7-minimal 2024-04-25T11:06:14.301 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:06:14.302 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-04-25T11:06:14.302 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T11:06:14.302 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T11:06:14.303 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T11:06:14.304 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-04-25T11:06:14.304 INFO:teuthology.orchestra.run.smithi157.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T11:06:14.304 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-04-25T11:06:14.304 INFO:teuthology.orchestra.run.smithi157.stdout: thermald 2024-04-25T11:06:14.343 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-04-25T11:06:14.343 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T11:06:14.343 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T11:06:14.344 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T11:06:14.345 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2024-04-25T11:06:14.345 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T11:06:14.526 INFO:teuthology.orchestra.run.smithi157.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T11:06:14.526 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 113 MB of archives. 2024-04-25T11:06:14.526 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T11:06:14.526 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T11:06:14.672 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:06:14.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T11:06:14.745 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:06:14.936 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-04-25T11:06:14.937 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T11:06:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: python2.7-minimal 2024-04-25T11:06:15.092 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:06:15.093 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-04-25T11:06:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T11:06:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T11:06:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T11:06:15.094 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-04-25T11:06:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T11:06:15.094 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-04-25T11:06:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: thermald 2024-04-25T11:06:15.140 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-04-25T11:06:15.140 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T11:06:15.140 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T11:06:15.140 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T11:06:15.142 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2024-04-25T11:06:15.142 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T11:06:15.321 INFO:teuthology.orchestra.run.smithi136.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T11:06:15.322 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 113 MB of archives. 2024-04-25T11:06:15.322 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T11:06:15.322 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T11:06:16.027 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T11:06:16.372 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T11:06:16.722 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T11:06:17.076 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T11:06:18.540 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T11:06:18.541 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T11:06:18.541 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T11:06:18.919 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T11:06:19.001 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T11:06:19.306 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T11:06:19.306 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T11:06:19.306 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T11:06:19.381 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-04-25T11:06:19.563 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T11:06:19.819 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T11:06:19.875 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T11:06:20.279 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 113 MB in 5s (24.0 MB/s) 2024-04-25T11:06:20.471 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T11:06:21.552 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T11:06:21.556 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:21.627 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:22.495 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T11:06:22.499 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:22.620 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:24.153 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T11:06:24.175 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:24.242 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:24.657 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T11:06:24.677 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:24.715 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:24.822 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T11:06:24.844 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:24.931 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:25.428 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T11:06:25.449 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:25.492 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:31.099 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:31.209 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:31.508 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:31.629 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:31.958 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T11:06:31.980 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T11:06:32.024 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T11:06:32.587 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:32.740 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:33.098 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:33.201 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:33.538 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T11:06:33.549 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T11:06:33.588 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T11:06:37.487 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T11:06:37.525 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:37.562 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:39.895 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T11:06:39.933 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T11:06:39.978 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:40.738 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:40.863 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:41.196 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T11:06:41.309 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:41.435 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T11:06:41.564 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:42.878 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:06:42.878 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T11:06:42.878 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:06:42.878 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T11:06:43.032 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:43.611 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T11:06:43.742 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T11:06:44.109 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T11:06:44.214 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:44.340 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T11:06:44.460 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:44.460 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:45.793 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:06:45.793 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T11:06:45.793 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:06:45.793 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T11:06:45.835 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T11:06:45.961 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T11:06:45.963 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:46.096 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:46.170 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T11:06:46.170 INFO:teuthology.orchestra.run.smithi157.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:06:47.376 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:49.201 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T11:06:49.352 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T11:06:49.478 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T11:06:49.548 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T11:06:49.548 INFO:teuthology.orchestra.run.smithi136.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:07:00.070 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T11:07:00.071 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub' 2024-04-25T11:07:00.088 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T11:07:00.112 INFO:teuthology.orchestra.run.smithi157.stdout:Generating grub configuration file ... 2024-04-25T11:07:00.423 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:07:00.440 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:07:00.767 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:07:00.773 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T11:07:00.836 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T11:07:00.842 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T11:07:01.009 INFO:teuthology.orchestra.run.smithi157.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T11:07:01.009 INFO:teuthology.orchestra.run.smithi157.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T11:07:01.009 INFO:teuthology.orchestra.run.smithi157.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T11:07:01.041 INFO:teuthology.orchestra.run.smithi157.stdout:done 2024-04-25T11:07:01.495 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-04-25T11:07:01.512 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-04-25T11:07:01.512 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-04-25T11:07:01.512 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-04-25T11:07:01.512 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-04-25T11:07:01.512 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.105.102 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-04-25T11:07:01.513 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-04-25T11:07:01.513 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T11:07:01.514 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T11:07:01.514 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T11:07:01.514 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-25T11:07:01.514 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T11:07:01.514 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-04-25T11:07:01.683 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-04-25T11:07:01.798 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T11:07:01.815 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:07:01.837 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T11:07:01.856 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T11:07:03.103 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:07:03.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T11:07:03.182 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:07:03.382 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-04-25T11:07:03.382 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-04-25T11:07:03.556 INFO:teuthology.orchestra.run.smithi157.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T11:07:03.557 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:07:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:07:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:07:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:07:03.557 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:07:03.558 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:07:03.558 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:07:03.599 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T11:07:03.601 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-04-25T11:07:03.612 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-04-25T11:07:03.612 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-04-25T11:07:03.612 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.105.102 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-04-25T11:07:03.613 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-04-25T11:07:03.614 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-04-25T11:07:03.657 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.HJMUtPVENZ 2024-04-25T11:07:03.657 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub/grub.cfg /tmp/tmp.HJMUtPVENZ 2024-04-25T11:07:03.714 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.HJMUtPVENZ 2024-04-25T11:07:03.843 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.HJMUtPVENZ is 10KB 2024-04-25T11:07:03.856 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.HJMUtPVENZ 2024-04-25T11:07:03.861 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T11:07:03.915 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:07:03.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T11:07:03.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T11:07:03.982 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T11:07:03.982 DEBUG:teuthology.orchestra.run.smithi157:> sudo update-grub 2024-04-25T11:07:04.125 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T11:07:04.125 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub' 2024-04-25T11:07:04.143 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T11:07:04.165 INFO:teuthology.orchestra.run.smithi136.stdout:Generating grub configuration file ... 2024-04-25T11:07:04.504 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:07:04.531 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:07:04.842 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:07:04.848 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T11:07:04.913 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T11:07:04.918 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T11:07:05.090 INFO:teuthology.orchestra.run.smithi136.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T11:07:05.090 INFO:teuthology.orchestra.run.smithi136.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T11:07:05.090 INFO:teuthology.orchestra.run.smithi136.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T11:07:05.138 INFO:teuthology.orchestra.run.smithi136.stdout:done 2024-04-25T11:07:05.147 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub' 2024-04-25T11:07:05.148 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T11:07:05.150 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-04-25T11:07:05.351 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:07:05.357 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:07:05.590 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:07:05.595 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T11:07:05.609 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-04-25T11:07:05.629 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.105.102 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-04-25T11:07:05.630 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-04-25T11:07:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-04-25T11:07:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-04-25T11:07:05.631 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T11:07:05.631 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T11:07:05.631 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T11:07:05.631 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T11:07:05.631 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T11:07:05.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-04-25T11:07:05.636 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T11:07:05.641 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T11:07:05.758 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T11:07:05.759 INFO:teuthology.orchestra.run.smithi157.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T11:07:05.759 INFO:teuthology.orchestra.run.smithi157.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T11:07:05.777 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-04-25T11:07:05.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-04-25T11:07:05.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-04-25T11:07:05.975 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T11:07:05.975 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:07:05.995 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T11:07:06.022 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T11:07:07.316 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:07:07.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T11:07:07.393 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:07:07.599 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-04-25T11:07:07.600 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:07:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:07:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:07:07.780 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:07:07.821 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T11:07:07.823 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-04-25T11:07:07.838 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.105.102 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-04-25T11:07:07.839 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-04-25T11:07:07.840 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-04-25T11:07:07.886 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.gGdZPs6eYZ 2024-04-25T11:07:07.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub/grub.cfg /tmp/tmp.gGdZPs6eYZ 2024-04-25T11:07:07.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.gGdZPs6eYZ 2024-04-25T11:07:08.057 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.gGdZPs6eYZ is 10KB 2024-04-25T11:07:08.069 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.gGdZPs6eYZ 2024-04-25T11:07:08.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T11:07:08.130 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:07:08.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T11:07:08.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T11:07:08.206 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T11:07:08.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo update-grub 2024-04-25T11:07:09.615 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub' 2024-04-25T11:07:09.616 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T11:07:09.619 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-04-25T11:07:09.807 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T11:07:09.813 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T11:07:10.079 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T11:07:10.084 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T11:07:10.127 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T11:07:10.133 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T11:07:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T11:07:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T11:07:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T11:07:10.272 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-04-25T11:07:10.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-04-25T11:07:35.793 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-25T11:07:35.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:07:35.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:07:40.294 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T11:07:40.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:07:40.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:07:54.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-04-25T11:07:58.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-25T11:08:03.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:08:03.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:06.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-04-25T11:08:07.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:08:07.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:10.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-25T11:08:18.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:08:18.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:21.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-04-25T11:08:22.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:08:22.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:36.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:08:36.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:41.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-25T11:08:54.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-04-25T11:08:56.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:08:56.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:08:56.442 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-25T11:08:57.045 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:08:57.046 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T11:08:57.046 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-04-25T11:08:57.094 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-105-generic 2024-04-25T11:08:57.094 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T11:08:57.094 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T11:08:57.094 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T11:08:58.095 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T11:08:58.095 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T11:08:58.159 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-04-25T11:08:58.183 DEBUG:teuthology.parallel:result is None 2024-04-25T11:09:12.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:09:12.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:09:13.290 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-04-25T11:09:14.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:09:14.056 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-25T11:09:14.056 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-04-25T11:09:14.109 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-105-generic 2024-04-25T11:09:14.109 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T11:09:14.109 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T11:09:14.109 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T11:09:15.110 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T11:09:15.111 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T11:09:15.194 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-04-25T11:09:15.210 DEBUG:teuthology.parallel:result is None 2024-04-25T11:09:15.210 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T11:09:15.219 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T11:09:15.219 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T11:09:15.222 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T11:09:15.227 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T11:09:15.234 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T11:09:15.243 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T11:09:15.243 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T11:09:15.270 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T11:09:15.465 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T11:09:15.471 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T11:09:15.471 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T11:09:15.474 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T11:09:15.503 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:09:15.504 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:09:15.512 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:09:15.513 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T11:09:15.515 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T11:09:15.522 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T11:09:15.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T11:09:15.557 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T11:09:15.571 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T11:09:15.581 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T11:09:15.581 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T11:09:15.614 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T11:09:15.619 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:09:15.668 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:09:15.716 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:09:15.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T11:09:15.820 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T11:09:15.825 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T11:09:15.872 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:09:15.873 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T11:09:15.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-04-25T11:09:15.937 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-04-25T11:09:16.007 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T11:09:16.017 INFO:teuthology.task.internal:Starting timer... 2024-04-25T11:09:16.017 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T11:09:16.028 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T11:09:16.036 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2024-04-25T11:09:16.036 DEBUG:teuthology.task.selinux:Excluding smithi157: OS 'ubuntu' does not support SELinux 2024-04-25T11:09:16.036 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T11:09:16.036 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T11:09:16.036 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T11:09:16.036 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T11:09:16.045 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T11:09:16.051 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T11:09:16.166 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T11:09:16.180 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T11:09:16.181 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T11:16:13.104 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-04-25T11:16:13.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:16:13.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:16:13.188 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-25T11:16:13.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-25T11:16:13.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:16:13.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T11:16:13.347 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-04-25T11:16:13.426 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-04-25T11:16:13.427 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T11:16:13.438 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T11:16:13.438 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T11:16:13.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:16:13.441 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T11:16:13.441 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Command line: ntpd -gq 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: ---------------------------------------------------- 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: corporation. Support and training for ntp-4 are 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: available at https://www.nwtime.org/support 2024-04-25T11:16:13.469 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: ---------------------------------------------------- 2024-04-25T11:16:13.470 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: proto: precision = 0.044 usec (-24) 2024-04-25T11:16:13.470 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: basedate set to 2022-02-04 2024-04-25T11:16:13.471 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: gps base set to 2022-02-06 (week 2196) 2024-04-25T11:16:13.471 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T11:16:13.471 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T11:16:13.471 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T11:16:13.472 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T11:16:13.472 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen normally on 4 lo [::1]:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2024-04-25T11:16:13.473 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:13 ntpd[17206]: Listening on routing socket on fd #22 for interface updates 2024-04-25T11:16:13.493 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Command line: ntpd -gq 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: ---------------------------------------------------- 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: corporation. Support and training for ntp-4 are 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: available at https://www.nwtime.org/support 2024-04-25T11:16:13.494 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: ---------------------------------------------------- 2024-04-25T11:16:13.495 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: proto: precision = 0.056 usec (-24) 2024-04-25T11:16:13.495 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: basedate set to 2022-02-04 2024-04-25T11:16:13.495 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: gps base set to 2022-02-06 (week 2196) 2024-04-25T11:16:13.495 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T11:16:13.495 INFO:teuthology.orchestra.run.smithi157.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T11:16:13.496 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T11:16:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen normally on 3 enp3s0f1 172.21.15.157:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen normally on 4 lo [::1]:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea1%5]:123 2024-04-25T11:16:13.497 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:13 ntpd[17196]: Listening on routing socket on fd #22 for interface updates 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:14.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:14 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:14 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:15 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:16.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:16 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:16.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:16 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:17.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:17.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:17 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:18 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:18.497 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:18.498 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:18 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:19.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:19.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:19.472 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:19 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:20.470 INFO:teuthology.orchestra.run.smithi136.stdout:25 Apr 11:16:20 ntpd[17206]: ntpd: time slew -0.000820 s 2024-04-25T11:16:20.471 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew -0.000820s 2024-04-25T11:16:20.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T11:16:20.471 INFO:teuthology.orchestra.run.smithi136.stderr:25 Apr 11:16:20 ntpd[17206]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T11:16:20.494 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stdout:25 Apr 11:16:20 ntpd[17196]: ntpd: time slew +0.000411 s 2024-04-25T11:16:20.495 INFO:teuthology.orchestra.run.smithi157.stdout:ntpd: time slew +0.000411s 2024-04-25T11:16:20.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T11:16:20.496 INFO:teuthology.orchestra.run.smithi157.stderr:25 Apr 11:16:20 ntpd[17196]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.590 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T11:16:20.591 INFO:teuthology.run_tasks:Running task install... 2024-04-25T11:16:20.601 DEBUG:teuthology.task.install:project ceph 2024-04-25T11:16:20.601 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T11:16:20.602 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T11:16:20.602 INFO:teuthology.task.install:Using flavor: default 2024-04-25T11:16:20.610 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T11:16:20.610 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T11:16:20.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2024-04-25T11:16:20.611 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-key list | grep Ceph 2024-04-25T11:16:20.666 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T11:16:20.693 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T11:16:20.693 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T11:16:20.693 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T11:16:20.694 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-25T11:16:20.694 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-25T11:16:20.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:16:20.729 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T11:16:20.729 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T11:16:20.729 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-25T11:16:20.729 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-25T11:16:20.729 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:16:20.839 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default/ 2024-04-25T11:16:20.839 INFO:teuthology.task.install.deb:Package version is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:16:20.966 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default/ 2024-04-25T11:16:20.966 INFO:teuthology.task.install.deb:Package version is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:16:21.119 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:16:21.119 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T11:16:21.133 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-04-25T11:16:21.230 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:16:21.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T11:16:21.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-04-25T11:16:21.311 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:16:21.326 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T11:16:21.348 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T11:16:21.421 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T11:16:21.433 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T11:16:21.436 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T11:16:21.448 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T11:16:21.455 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T11:16:21.492 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T11:16:21.529 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T11:16:21.537 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T11:16:21.558 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T11:16:21.584 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T11:16:21.606 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T11:16:21.613 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-25T11:16:21.627 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2024-04-25T11:16:21.655 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-25T11:16:21.666 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2024-04-25T11:16:21.755 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 29.2 kB in 0s (68.8 kB/s) 2024-04-25T11:16:21.799 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 29.2 kB in 0s (77.6 kB/s) 2024-04-25T11:16:22.749 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:16:22.766 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3291-g994e2f20-1jammy cephadm=19.0.0-3291-g994e2f20-1jammy ceph-mds=19.0.0-3291-g994e2f20-1jammy ceph-mgr=19.0.0-3291-g994e2f20-1jammy ceph-common=19.0.0-3291-g994e2f20-1jammy ceph-fuse=19.0.0-3291-g994e2f20-1jammy ceph-test=19.0.0-3291-g994e2f20-1jammy ceph-volume=19.0.0-3291-g994e2f20-1jammy radosgw=19.0.0-3291-g994e2f20-1jammy python3-rados=19.0.0-3291-g994e2f20-1jammy python3-rgw=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy python3-rbd=19.0.0-3291-g994e2f20-1jammy libcephfs2=19.0.0-3291-g994e2f20-1jammy libcephfs-dev=19.0.0-3291-g994e2f20-1jammy librados2=19.0.0-3291-g994e2f20-1jammy librbd1=19.0.0-3291-g994e2f20-1jammy rbd-fuse=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy cephfs-shell=19.0.0-3291-g994e2f20-1jammy cephfs-top=19.0.0-3291-g994e2f20-1jammy cephfs-mirror=19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:16:22.766 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:16:22.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3291-g994e2f20-1jammy cephadm=19.0.0-3291-g994e2f20-1jammy ceph-mds=19.0.0-3291-g994e2f20-1jammy ceph-mgr=19.0.0-3291-g994e2f20-1jammy ceph-common=19.0.0-3291-g994e2f20-1jammy ceph-fuse=19.0.0-3291-g994e2f20-1jammy ceph-test=19.0.0-3291-g994e2f20-1jammy ceph-volume=19.0.0-3291-g994e2f20-1jammy radosgw=19.0.0-3291-g994e2f20-1jammy python3-rados=19.0.0-3291-g994e2f20-1jammy python3-rgw=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy python3-rbd=19.0.0-3291-g994e2f20-1jammy libcephfs2=19.0.0-3291-g994e2f20-1jammy libcephfs-dev=19.0.0-3291-g994e2f20-1jammy librados2=19.0.0-3291-g994e2f20-1jammy librbd1=19.0.0-3291-g994e2f20-1jammy rbd-fuse=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy cephfs-shell=19.0.0-3291-g994e2f20-1jammy cephfs-top=19.0.0-3291-g994e2f20-1jammy cephfs-mirror=19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:16:22.818 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:16:22.841 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:16:22.995 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-04-25T11:16:22.996 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-04-25T11:16:23.039 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-04-25T11:16:23.040 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-04-25T11:16:23.171 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:16:23.172 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:16:23.172 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:16:23.172 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:16:23.172 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T11:16:23.173 INFO:teuthology.orchestra.run.smithi157.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T11:16:23.174 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T11:16:23.174 INFO:teuthology.orchestra.run.smithi157.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T11:16:23.174 INFO:teuthology.orchestra.run.smithi157.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T11:16:23.174 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T11:16:23.174 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-25T11:16:23.175 INFO:teuthology.orchestra.run.smithi157.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T11:16:23.176 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-04-25T11:16:23.177 INFO:teuthology.orchestra.run.smithi157.stdout: btrfs-tools 2024-04-25T11:16:23.223 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:16:23.224 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:16:23.225 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-04-25T11:16:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-25T11:16:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T11:16:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-04-25T11:16:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-25T11:16:23.281 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-25T11:16:23.283 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-25T11:16:23.284 INFO:teuthology.orchestra.run.smithi157.stdout: socat unzip xmlstarlet zip 2024-04-25T11:16:23.332 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-04-25T11:16:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T11:16:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T11:16:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T11:16:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-25T11:16:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: socat unzip xmlstarlet zip 2024-04-25T11:16:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T11:16:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 175 MB of archives. 2024-04-25T11:16:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-25T11:16:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-25T11:16:23.457 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T11:16:23.472 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T11:16:23.472 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 175 MB of archives. 2024-04-25T11:16:23.472 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-25T11:16:23.472 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-25T11:16:23.484 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-25T11:16:23.554 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T11:16:23.598 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T11:16:23.600 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-25T11:16:23.643 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3291-g994e2f20-1jammy [3,525 kB] 2024-04-25T11:16:23.678 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T11:16:23.688 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T11:16:23.733 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T11:16:23.740 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3291-g994e2f20-1jammy [3,525 kB] 2024-04-25T11:16:23.796 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T11:16:23.894 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T11:16:23.907 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T11:16:23.941 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T11:16:23.946 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T11:16:23.959 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T11:16:23.995 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T11:16:24.002 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T11:16:24.004 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T11:16:24.021 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T11:16:24.032 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T11:16:24.040 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T11:16:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T11:16:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T11:16:24.051 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T11:16:24.055 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T11:16:24.081 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T11:16:24.082 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T11:16:24.083 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T11:16:24.085 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T11:16:24.085 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T11:16:24.086 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T11:16:24.087 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T11:16:24.088 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T11:16:24.116 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T11:16:24.117 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T11:16:24.133 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T11:16:24.142 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T11:16:24.144 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T11:16:24.145 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T11:16:24.146 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T11:16:24.147 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T11:16:24.148 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T11:16:24.149 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T11:16:24.150 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T11:16:24.152 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T11:16:24.170 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T11:16:24.178 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T11:16:24.182 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T11:16:24.183 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T11:16:24.220 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T11:16:24.221 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T11:16:24.222 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T11:16:24.223 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T11:16:24.226 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T11:16:24.230 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T11:16:24.232 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T11:16:24.234 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T11:16:24.251 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T11:16:24.256 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T11:16:24.280 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T11:16:24.281 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T11:16:24.286 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T11:16:24.343 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T11:16:24.374 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T11:16:24.376 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T11:16:24.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T11:16:24.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T11:16:24.381 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T11:16:24.413 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T11:16:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T11:16:24.419 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T11:16:24.450 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T11:16:24.450 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T11:16:24.463 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T11:16:24.464 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T11:16:24.464 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T11:16:24.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T11:16:24.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T11:16:24.467 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T11:16:24.471 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T11:16:24.486 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T11:16:24.488 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T11:16:24.489 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T11:16:24.507 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T11:16:24.512 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-25T11:16:24.512 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T11:16:24.514 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T11:16:24.514 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T11:16:24.521 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T11:16:24.522 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T11:16:24.534 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T11:16:24.534 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T11:16:24.535 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T11:16:24.556 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T11:16:24.558 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T11:16:24.559 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T11:16:24.569 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T11:16:24.570 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T11:16:24.570 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T11:16:24.573 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T11:16:24.587 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T11:16:24.915 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T11:16:24.930 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T11:16:24.946 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T11:16:24.954 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T11:16:24.955 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T11:16:24.972 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T11:16:25.009 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T11:16:25.022 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T11:16:25.026 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T11:16:25.027 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T11:16:25.197 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T11:16:25.281 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T11:16:25.287 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T11:16:25.362 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T11:16:25.388 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T11:16:25.404 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T11:16:25.421 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T11:16:25.613 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T11:16:25.629 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T11:16:25.636 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T11:16:25.641 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T11:16:25.824 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T11:16:25.907 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T11:16:25.923 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T11:16:26.108 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T11:16:26.159 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T11:16:26.173 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T11:16:26.331 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T11:16:26.332 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T11:16:26.337 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T11:16:26.342 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T11:16:26.356 INFO:teuthology.orchestra.run.smithi157.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T11:16:26.541 INFO:teuthology.orchestra.run.smithi157.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T11:16:26.688 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3291-g994e2f20-1jammy [3,287 kB] 2024-04-25T11:16:26.804 INFO:teuthology.orchestra.run.smithi157.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T11:16:26.919 INFO:teuthology.orchestra.run.smithi157.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T11:16:26.964 INFO:teuthology.orchestra.run.smithi157.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T11:16:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T11:16:27.059 INFO:teuthology.orchestra.run.smithi157.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T11:16:27.071 INFO:teuthology.orchestra.run.smithi157.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T11:16:27.112 INFO:teuthology.orchestra.run.smithi157.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T11:16:27.266 INFO:teuthology.orchestra.run.smithi157.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3291-g994e2f20-1jammy [3,287 kB] 2024-04-25T11:16:27.373 INFO:teuthology.orchestra.run.smithi157.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T11:16:27.375 INFO:teuthology.orchestra.run.smithi157.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T11:16:27.377 INFO:teuthology.orchestra.run.smithi157.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T11:16:27.541 INFO:teuthology.orchestra.run.smithi157.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T11:16:27.876 INFO:teuthology.orchestra.run.smithi157.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T11:16:28.270 INFO:teuthology.orchestra.run.smithi157.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T11:16:28.280 INFO:teuthology.orchestra.run.smithi157.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T11:16:28.281 INFO:teuthology.orchestra.run.smithi157.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T11:16:28.352 INFO:teuthology.orchestra.run.smithi157.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T11:16:28.353 INFO:teuthology.orchestra.run.smithi157.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T11:16:28.354 INFO:teuthology.orchestra.run.smithi157.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T11:16:28.355 INFO:teuthology.orchestra.run.smithi157.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T11:16:28.357 INFO:teuthology.orchestra.run.smithi157.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T11:16:28.391 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3291-g994e2f20-1jammy [726 kB] 2024-04-25T11:16:28.522 INFO:teuthology.orchestra.run.smithi157.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T11:16:28.620 INFO:teuthology.orchestra.run.smithi157.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T11:16:28.714 INFO:teuthology.orchestra.run.smithi157.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T11:16:28.938 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3291-g994e2f20-1jammy [356 kB] 2024-04-25T11:16:28.959 INFO:teuthology.orchestra.run.smithi157.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T11:16:29.001 INFO:teuthology.orchestra.run.smithi157.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T11:16:29.026 INFO:teuthology.orchestra.run.smithi157.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T11:16:29.034 INFO:teuthology.orchestra.run.smithi157.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T11:16:29.078 INFO:teuthology.orchestra.run.smithi157.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T11:16:29.103 INFO:teuthology.orchestra.run.smithi157.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-25T11:16:29.105 INFO:teuthology.orchestra.run.smithi157.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T11:16:29.112 INFO:teuthology.orchestra.run.smithi157.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T11:16:29.254 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3291-g994e2f20-1jammy [32.8 kB] 2024-04-25T11:16:29.279 INFO:teuthology.orchestra.run.smithi157.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T11:16:29.281 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3291-g994e2f20-1jammy [180 kB] 2024-04-25T11:16:29.415 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3291-g994e2f20-1jammy [67.1 kB] 2024-04-25T11:16:29.473 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3291-g994e2f20-1jammy [330 kB] 2024-04-25T11:16:29.676 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3291-g994e2f20-1jammy [7,189 kB] 2024-04-25T11:16:29.816 INFO:teuthology.orchestra.run.smithi157.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3291-g994e2f20-1jammy [726 kB] 2024-04-25T11:16:29.817 INFO:teuthology.orchestra.run.smithi157.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T11:16:29.822 INFO:teuthology.orchestra.run.smithi157.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T11:16:29.862 INFO:teuthology.orchestra.run.smithi157.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T11:16:29.863 INFO:teuthology.orchestra.run.smithi157.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T11:16:29.869 INFO:teuthology.orchestra.run.smithi157.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T11:16:29.871 INFO:teuthology.orchestra.run.smithi157.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T11:16:29.913 INFO:teuthology.orchestra.run.smithi157.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T11:16:29.922 INFO:teuthology.orchestra.run.smithi157.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T11:16:29.994 INFO:teuthology.orchestra.run.smithi157.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T11:16:29.995 INFO:teuthology.orchestra.run.smithi157.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T11:16:30.159 INFO:teuthology.orchestra.run.smithi157.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T11:16:30.247 INFO:teuthology.orchestra.run.smithi157.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3291-g994e2f20-1jammy [356 kB] 2024-04-25T11:16:30.501 INFO:teuthology.orchestra.run.smithi157.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3291-g994e2f20-1jammy [32.8 kB] 2024-04-25T11:16:30.503 INFO:teuthology.orchestra.run.smithi157.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3291-g994e2f20-1jammy [180 kB] 2024-04-25T11:16:30.602 INFO:teuthology.orchestra.run.smithi157.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T11:16:30.618 INFO:teuthology.orchestra.run.smithi157.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T11:16:30.644 INFO:teuthology.orchestra.run.smithi157.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3291-g994e2f20-1jammy [67.1 kB] 2024-04-25T11:16:30.651 INFO:teuthology.orchestra.run.smithi157.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3291-g994e2f20-1jammy [330 kB] 2024-04-25T11:16:30.865 INFO:teuthology.orchestra.run.smithi157.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3291-g994e2f20-1jammy [7,189 kB] 2024-04-25T11:16:33.188 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3291-g994e2f20-1jammy [112 kB] 2024-04-25T11:16:33.246 INFO:teuthology.orchestra.run.smithi136.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3291-g994e2f20-1jammy [471 kB] 2024-04-25T11:16:33.457 INFO:teuthology.orchestra.run.smithi136.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3291-g994e2f20-1jammy [25.6 MB] 2024-04-25T11:16:34.402 INFO:teuthology.orchestra.run.smithi157.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3291-g994e2f20-1jammy [112 kB] 2024-04-25T11:16:34.461 INFO:teuthology.orchestra.run.smithi157.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3291-g994e2f20-1jammy [471 kB] 2024-04-25T11:16:34.609 INFO:teuthology.orchestra.run.smithi157.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3291-g994e2f20-1jammy [25.6 MB] 2024-04-25T11:16:39.413 INFO:teuthology.orchestra.run.smithi136.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3291-g994e2f20-1jammy [4,917 kB] 2024-04-25T11:16:39.962 INFO:teuthology.orchestra.run.smithi157.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3291-g994e2f20-1jammy [4,917 kB] 2024-04-25T11:16:40.132 INFO:teuthology.orchestra.run.smithi136.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3291-g994e2f20-1jammy [244 kB] 2024-04-25T11:16:40.168 INFO:teuthology.orchestra.run.smithi136.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3291-g994e2f20-1jammy [125 kB] 2024-04-25T11:16:40.188 INFO:teuthology.orchestra.run.smithi136.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3291-g994e2f20-1jammy [1,542 kB] 2024-04-25T11:16:40.415 INFO:teuthology.orchestra.run.smithi136.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3291-g994e2f20-1jammy [6,148 kB] 2024-04-25T11:16:40.650 INFO:teuthology.orchestra.run.smithi157.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3291-g994e2f20-1jammy [244 kB] 2024-04-25T11:16:40.686 INFO:teuthology.orchestra.run.smithi157.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3291-g994e2f20-1jammy [125 kB] 2024-04-25T11:16:40.693 INFO:teuthology.orchestra.run.smithi157.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3291-g994e2f20-1jammy [1,542 kB] 2024-04-25T11:16:40.917 INFO:teuthology.orchestra.run.smithi157.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3291-g994e2f20-1jammy [6,148 kB] 2024-04-25T11:16:41.298 INFO:teuthology.orchestra.run.smithi136.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3291-g994e2f20-1jammy [22.5 MB] 2024-04-25T11:16:41.800 INFO:teuthology.orchestra.run.smithi157.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3291-g994e2f20-1jammy [22.5 MB] 2024-04-25T11:16:44.353 INFO:teuthology.orchestra.run.smithi136.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3291-g994e2f20-1jammy [14.1 kB] 2024-04-25T11:16:44.353 INFO:teuthology.orchestra.run.smithi136.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3291-g994e2f20-1jammy [856 kB] 2024-04-25T11:16:44.393 INFO:teuthology.orchestra.run.smithi136.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3291-g994e2f20-1jammy [2,367 kB] 2024-04-25T11:16:44.692 INFO:teuthology.orchestra.run.smithi136.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3291-g994e2f20-1jammy [791 kB] 2024-04-25T11:16:44.774 INFO:teuthology.orchestra.run.smithi136.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3291-g994e2f20-1jammy [148 kB] 2024-04-25T11:16:44.806 INFO:teuthology.orchestra.run.smithi136.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3291-g994e2f20-1jammy [2,297 kB] 2024-04-25T11:16:44.813 INFO:teuthology.orchestra.run.smithi157.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3291-g994e2f20-1jammy [14.1 kB] 2024-04-25T11:16:44.814 INFO:teuthology.orchestra.run.smithi157.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3291-g994e2f20-1jammy [856 kB] 2024-04-25T11:16:44.856 INFO:teuthology.orchestra.run.smithi157.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3291-g994e2f20-1jammy [2,367 kB] 2024-04-25T11:16:45.026 INFO:teuthology.orchestra.run.smithi136.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3291-g994e2f20-1jammy [8,625 kB] 2024-04-25T11:16:45.077 INFO:teuthology.orchestra.run.smithi157.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3291-g994e2f20-1jammy [791 kB] 2024-04-25T11:16:45.149 INFO:teuthology.orchestra.run.smithi157.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3291-g994e2f20-1jammy [148 kB] 2024-04-25T11:16:45.179 INFO:teuthology.orchestra.run.smithi157.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3291-g994e2f20-1jammy [2,297 kB] 2024-04-25T11:16:45.400 INFO:teuthology.orchestra.run.smithi157.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3291-g994e2f20-1jammy [8,625 kB] 2024-04-25T11:16:45.837 INFO:teuthology.orchestra.run.smithi136.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3291-g994e2f20-1jammy [14.2 kB] 2024-04-25T11:16:45.838 INFO:teuthology.orchestra.run.smithi136.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3291-g994e2f20-1jammy [44.9 MB] 2024-04-25T11:16:46.158 INFO:teuthology.orchestra.run.smithi157.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3291-g994e2f20-1jammy [14.2 kB] 2024-04-25T11:16:46.158 INFO:teuthology.orchestra.run.smithi157.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3291-g994e2f20-1jammy [44.9 MB] 2024-04-25T11:16:49.340 INFO:teuthology.orchestra.run.smithi136.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3291-g994e2f20-1jammy [124 kB] 2024-04-25T11:16:49.342 INFO:teuthology.orchestra.run.smithi136.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3291-g994e2f20-1jammy [236 kB] 2024-04-25T11:16:49.347 INFO:teuthology.orchestra.run.smithi136.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3291-g994e2f20-1jammy [33.9 kB] 2024-04-25T11:16:49.348 INFO:teuthology.orchestra.run.smithi136.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3291-g994e2f20-1jammy [23.4 kB] 2024-04-25T11:16:49.349 INFO:teuthology.orchestra.run.smithi136.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3291-g994e2f20-1jammy [39.0 kB] 2024-04-25T11:16:49.350 INFO:teuthology.orchestra.run.smithi136.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3291-g994e2f20-1jammy [14.2 MB] 2024-04-25T11:16:49.548 INFO:teuthology.orchestra.run.smithi157.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3291-g994e2f20-1jammy [124 kB] 2024-04-25T11:16:49.551 INFO:teuthology.orchestra.run.smithi157.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3291-g994e2f20-1jammy [236 kB] 2024-04-25T11:16:49.556 INFO:teuthology.orchestra.run.smithi157.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3291-g994e2f20-1jammy [33.9 kB] 2024-04-25T11:16:49.557 INFO:teuthology.orchestra.run.smithi157.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3291-g994e2f20-1jammy [23.4 kB] 2024-04-25T11:16:49.558 INFO:teuthology.orchestra.run.smithi157.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3291-g994e2f20-1jammy [39.0 kB] 2024-04-25T11:16:49.559 INFO:teuthology.orchestra.run.smithi157.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3291-g994e2f20-1jammy [14.2 MB] 2024-04-25T11:16:50.171 INFO:teuthology.orchestra.run.smithi136.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3291-g994e2f20-1jammy [92.6 kB] 2024-04-25T11:16:50.354 INFO:teuthology.orchestra.run.smithi157.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3291-g994e2f20-1jammy [92.6 kB] 2024-04-25T11:16:50.793 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 175 MB in 27s (6,515 kB/s) 2024-04-25T11:16:50.933 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-25T11:16:50.977 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 175 MB in 27s (6,459 kB/s) 2024-04-25T11:16:51.029 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T11:16:51.035 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-25T11:16:51.073 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-25T11:16:51.258 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-25T11:16:51.360 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T11:16:51.366 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-25T11:16:51.367 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T11:16:51.394 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T11:16:51.414 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-25T11:16:51.434 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T11:16:51.667 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T11:16:51.693 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T11:16:51.725 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T11:16:51.728 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cmd2. 2024-04-25T11:16:51.755 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-25T11:16:51.786 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-25T11:16:52.036 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cmd2. 2024-04-25T11:16:52.062 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-25T11:16:52.082 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T11:16:52.103 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-25T11:16:52.108 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T11:16:52.173 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T11:16:52.481 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T11:16:52.507 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T11:16:52.569 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T11:16:52.589 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T11:16:52.595 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T11:16:52.635 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T11:16:52.952 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T11:16:52.978 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T11:16:52.997 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T11:16:53.018 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T11:16:53.023 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T11:16:53.063 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T11:16:53.338 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T11:16:53.364 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T11:16:53.396 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T11:16:53.442 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T11:16:53.469 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:53.508 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:53.841 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T11:16:53.867 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:53.887 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T11:16:53.913 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:53.916 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:53.953 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:54.290 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T11:16:54.317 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:54.319 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T11:16:54.345 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:54.356 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:54.386 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:54.668 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T11:16:54.694 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T11:16:54.697 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T11:16:54.712 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T11:16:54.726 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T11:16:54.747 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:16:55.012 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librados2. 2024-04-25T11:16:55.027 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../010-librados2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:55.070 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:55.109 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T11:16:55.135 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T11:16:55.175 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T11:16:55.487 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librados2. 2024-04-25T11:16:55.513 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../010-librados2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:55.548 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librbd1. 2024-04-25T11:16:55.553 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:55.575 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../011-librbd1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:55.618 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:56.056 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librbd1. 2024-04-25T11:16:56.082 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../011-librbd1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:56.122 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:56.243 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T11:16:56.258 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:56.292 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:57.288 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2024-04-25T11:16:57.314 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:57.354 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:57.451 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T11:16:57.478 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:57.518 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:57.673 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T11:16:57.700 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:16:57.740 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:57.812 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rados. 2024-04-25T11:16:57.838 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:57.878 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:57.934 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T11:16:57.960 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:57.992 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:58.165 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T11:16:58.191 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:16:58.223 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:58.303 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T11:16:58.330 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:16:58.370 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:58.451 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T11:16:58.466 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:58.509 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:58.673 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T11:16:58.699 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T11:16:58.740 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T11:16:58.820 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T11:16:58.846 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:16:58.886 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:58.992 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T11:16:59.019 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:59.059 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:59.231 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T11:16:59.258 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T11:16:59.298 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T11:16:59.362 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T11:16:59.388 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T11:16:59.420 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T11:16:59.534 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T11:16:59.549 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:16:59.584 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:16:59.698 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T11:16:59.724 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T11:16:59.765 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T11:16:59.920 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T11:16:59.946 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T11:16:59.987 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T11:17:00.067 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T11:17:00.090 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T11:17:00.134 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T11:17:00.264 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T11:17:00.290 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T11:17:00.331 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T11:17:00.421 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua5.1. 2024-04-25T11:17:00.446 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T11:17:00.487 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T11:17:00.634 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T11:17:00.649 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T11:17:00.692 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T11:17:00.765 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-any. 2024-04-25T11:17:00.780 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-25T11:17:00.823 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T11:17:00.987 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua5.1. 2024-04-25T11:17:01.002 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T11:17:01.037 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T11:17:01.059 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package zip. 2024-04-25T11:17:01.074 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-25T11:17:01.117 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T11:17:01.340 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-any. 2024-04-25T11:17:01.366 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-25T11:17:01.415 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T11:17:01.438 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package unzip. 2024-04-25T11:17:01.464 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T11:17:01.504 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T11:17:01.643 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package zip. 2024-04-25T11:17:01.658 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-25T11:17:01.701 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T11:17:01.875 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package luarocks. 2024-04-25T11:17:01.901 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T11:17:01.941 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T11:17:02.291 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2024-04-25T11:17:02.296 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package unzip. 2024-04-25T11:17:02.307 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../027-librgw2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:02.322 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T11:17:02.341 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:02.387 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T11:17:02.792 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package luarocks. 2024-04-25T11:17:02.817 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T11:17:02.858 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T11:17:02.927 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T11:17:02.942 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:02.985 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:03.241 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librgw2. 2024-04-25T11:17:03.267 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../027-librgw2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:03.307 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:03.313 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T11:17:03.339 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T11:17:03.380 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T11:17:03.708 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T11:17:03.734 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:03.774 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:03.985 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T11:17:04.012 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:04.060 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:04.102 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2024-04-25T11:17:04.129 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:04.160 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:04.430 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T11:17:04.456 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T11:17:04.496 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T11:17:04.791 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T11:17:04.817 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:04.857 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:05.369 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-common. 2024-04-25T11:17:05.396 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:05.444 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:05.683 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2024-04-25T11:17:05.709 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:05.758 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:06.537 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T11:17:06.563 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T11:17:06.603 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T11:17:06.848 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T11:17:06.874 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T11:17:06.915 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T11:17:07.041 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-base. 2024-04-25T11:17:07.067 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:07.116 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:07.268 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T11:17:07.283 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T11:17:07.326 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T11:17:07.629 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T11:17:07.655 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T11:17:07.697 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T11:17:07.812 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T11:17:07.827 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T11:17:07.862 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T11:17:07.950 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T11:17:07.976 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T11:17:08.008 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T11:17:08.148 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T11:17:08.174 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T11:17:08.215 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T11:17:08.295 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T11:17:08.320 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T11:17:08.361 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T11:17:08.543 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T11:17:08.569 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T11:17:08.609 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T11:17:08.647 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-portend. 2024-04-25T11:17:08.673 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-25T11:17:08.722 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T11:17:08.870 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T11:17:08.886 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T11:17:08.929 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T11:17:09.000 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T11:17:09.027 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T11:17:09.058 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T11:17:09.257 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T11:17:09.283 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T11:17:09.323 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T11:17:09.336 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T11:17:09.363 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T11:17:09.403 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T11:17:09.635 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T11:17:09.661 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T11:17:09.702 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T11:17:09.757 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T11:17:09.783 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T11:17:09.823 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T11:17:09.979 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-portend. 2024-04-25T11:17:10.005 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-25T11:17:10.037 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T11:17:10.110 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T11:17:10.136 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T11:17:10.176 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T11:17:10.323 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T11:17:10.349 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T11:17:10.390 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T11:17:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T11:17:10.439 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T11:17:10.479 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T11:17:10.685 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T11:17:10.712 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T11:17:10.751 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T11:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2024-04-25T11:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T11:17:10.890 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T11:17:11.080 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T11:17:11.106 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T11:17:11.146 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T11:17:11.194 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T11:17:11.221 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T11:17:11.252 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T11:17:11.458 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T11:17:11.484 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T11:17:11.524 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T11:17:11.530 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T11:17:11.556 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T11:17:11.596 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T11:17:11.836 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T11:17:11.862 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T11:17:11.891 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2024-04-25T11:17:11.902 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T11:17:11.917 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T11:17:11.957 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T11:17:12.172 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-mako. 2024-04-25T11:17:12.198 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T11:17:12.230 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T11:17:12.302 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T11:17:12.328 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T11:17:12.372 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T11:17:12.550 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T11:17:12.576 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T11:17:12.616 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T11:17:12.688 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T11:17:12.704 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T11:17:12.755 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T11:17:13.000 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2024-04-25T11:17:13.026 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T11:17:13.066 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T11:17:13.080 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T11:17:13.107 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T11:17:13.138 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T11:17:13.378 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T11:17:13.393 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T11:17:13.427 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T11:17:13.441 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webob. 2024-04-25T11:17:13.467 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T11:17:13.510 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T11:17:13.664 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T11:17:13.690 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T11:17:13.731 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T11:17:13.852 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T11:17:13.878 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T11:17:13.922 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T11:17:14.034 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T11:17:14.060 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T11:17:14.100 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T11:17:14.230 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T11:17:14.257 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T11:17:14.296 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T11:17:14.428 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T11:17:14.454 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T11:17:14.494 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T11:17:14.583 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-paste. 2024-04-25T11:17:14.609 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T11:17:14.649 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T11:17:14.850 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T11:17:14.876 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T11:17:14.917 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T11:17:14.944 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T11:17:14.959 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T11:17:14.994 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T11:17:15.202 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-idna. 2024-04-25T11:17:15.229 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-25T11:17:15.260 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T11:17:15.297 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T11:17:15.324 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T11:17:15.363 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T11:17:15.538 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T11:17:15.565 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T11:17:15.605 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T11:17:15.692 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T11:17:15.718 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T11:17:15.758 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T11:17:15.884 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests. 2024-04-25T11:17:15.910 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T11:17:15.950 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T11:17:16.069 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T11:17:16.096 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T11:17:16.136 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T11:17:16.261 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T11:17:16.288 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T11:17:16.327 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T11:17:16.499 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T11:17:16.526 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T11:17:16.565 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T11:17:16.589 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T11:17:16.605 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:16.839 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:16.852 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-idna. 2024-04-25T11:17:16.878 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-25T11:17:16.918 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T11:17:17.211 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T11:17:17.238 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:17.238 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T11:17:17.264 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T11:17:17.277 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:17.304 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T11:17:17.530 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T11:17:17.545 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T11:17:17.588 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T11:17:17.624 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests. 2024-04-25T11:17:17.651 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T11:17:17.691 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T11:17:17.924 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T11:17:17.950 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T11:17:17.991 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T11:17:18.136 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T11:17:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T11:17:18.269 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T11:17:18.296 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:18.327 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:18.346 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T11:17:18.639 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T11:17:18.665 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:18.681 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T11:17:18.697 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:18.707 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:18.747 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:19.102 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T11:17:19.128 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:19.168 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:19.359 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T11:17:19.386 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:19.425 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:19.471 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T11:17:19.498 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T11:17:19.538 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T11:17:19.807 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T11:17:19.834 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T11:17:19.874 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T11:17:20.210 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T11:17:20.237 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:20.277 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:20.454 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2024-04-25T11:17:20.480 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../068-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:20.520 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:20.680 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T11:17:20.706 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:20.746 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:20.772 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T11:17:20.799 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:20.849 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:21.226 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T11:17:21.252 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:21.291 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T11:17:21.292 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:21.318 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:21.358 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:21.720 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2024-04-25T11:17:21.747 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../071-cephadm_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:21.786 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:22.073 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T11:17:22.099 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T11:17:22.139 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T11:17:22.434 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T11:17:22.436 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph. 2024-04-25T11:17:22.449 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T11:17:22.462 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../068-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:22.492 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:22.502 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:22.855 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T11:17:22.882 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:22.922 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:23.275 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T11:17:23.301 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:23.333 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:23.472 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-babel. 2024-04-25T11:17:23.499 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T11:17:23.538 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:23.728 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephadm. 2024-04-25T11:17:23.754 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../071-cephadm_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:23.794 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:23.867 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T11:17:23.893 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T11:17:23.933 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T11:17:24.114 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T11:17:24.140 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T11:17:24.180 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T11:17:24.278 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T11:17:24.304 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:24.508 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T11:17:24.523 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T11:17:24.566 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:24.681 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T11:17:24.707 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T11:17:24.756 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T11:17:25.059 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2024-04-25T11:17:25.085 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T11:17:25.134 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T11:17:25.445 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T11:17:25.472 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:25.511 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:25.655 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-babel. 2024-04-25T11:17:25.681 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T11:17:25.721 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:26.049 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T11:17:26.076 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T11:17:26.115 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T11:17:26.468 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T11:17:26.495 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:26.535 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:26.783 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T11:17:26.800 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T11:17:26.833 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T11:17:26.880 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T11:17:26.907 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T11:17:26.946 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T11:17:27.224 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-routes. 2024-04-25T11:17:27.240 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T11:17:27.274 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T11:17:27.370 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T11:17:27.386 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T11:17:27.428 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T11:17:27.594 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T11:17:27.621 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:27.660 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:27.782 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T11:17:27.809 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T11:17:27.934 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T11:17:28.220 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T11:17:28.247 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T11:17:28.286 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T11:17:29.038 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T11:17:29.065 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:29.095 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:29.132 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T11:17:29.160 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T11:17:29.199 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T11:17:29.702 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T11:17:29.729 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T11:17:29.769 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T11:17:30.057 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T11:17:30.083 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T11:17:30.122 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T11:17:30.123 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T11:17:30.149 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T11:17:30.283 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T11:17:30.427 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T11:17:30.453 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T11:17:30.494 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T11:17:30.569 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T11:17:30.596 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T11:17:30.635 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T11:17:30.847 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T11:17:30.874 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T11:17:30.913 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T11:17:31.200 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T11:17:31.215 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-25T11:17:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T11:17:31.378 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T11:17:31.405 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:31.444 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:31.511 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T11:17:31.538 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T11:17:31.578 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T11:17:31.898 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T11:17:31.925 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T11:17:31.964 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T11:17:32.284 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T11:17:32.311 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T11:17:32.350 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T11:17:32.381 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T11:17:32.408 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T11:17:32.447 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T11:17:32.750 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T11:17:32.777 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T11:17:32.779 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T11:17:32.806 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T11:17:32.816 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T11:17:32.845 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T11:17:33.162 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T11:17:33.189 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T11:17:33.253 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T11:17:33.526 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T11:17:33.542 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:33.564 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T11:17:33.584 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:33.592 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-25T11:17:33.631 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T11:17:33.904 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T11:17:33.931 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T11:17:33.943 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T11:17:33.970 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T11:17:33.970 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T11:17:34.001 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T11:17:34.257 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T11:17:34.284 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T11:17:34.315 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T11:17:34.363 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T11:17:34.390 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T11:17:34.438 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T11:17:34.576 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2024-04-25T11:17:34.593 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T11:17:34.634 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T11:17:34.816 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T11:17:34.843 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T11:17:34.882 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T11:17:34.887 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2024-04-25T11:17:34.914 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T11:17:34.953 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T11:17:35.219 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T11:17:35.246 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T11:17:35.265 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T11:17:35.286 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T11:17:35.292 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T11:17:35.332 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T11:17:35.711 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2024-04-25T11:17:35.738 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:35.777 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:35.983 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T11:17:35.999 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:36.041 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:36.370 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T11:17:36.396 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T11:17:36.436 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T11:17:36.714 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T11:17:36.741 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T11:17:36.772 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T11:17:37.033 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package jq. 2024-04-25T11:17:37.060 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T11:17:37.099 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T11:17:37.352 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package socat. 2024-04-25T11:17:37.379 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T11:17:37.418 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T11:17:37.755 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T11:17:37.771 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T11:17:37.813 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T11:17:38.074 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T11:17:38.101 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:38.140 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:38.168 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-test. 2024-04-25T11:17:38.188 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:38.226 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:38.486 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-25T11:17:38.513 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:38.552 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:38.931 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-colorama. 2024-04-25T11:17:38.958 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-25T11:17:38.997 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-25T11:17:39.275 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-shell. 2024-04-25T11:17:39.302 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:39.341 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:39.602 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-top. 2024-04-25T11:17:39.629 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:39.668 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:39.888 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T11:17:39.915 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:39.946 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:40.257 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T11:17:40.285 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T11:17:40.324 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T11:17:40.603 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T11:17:40.619 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T11:17:40.653 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T11:17:40.722 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T11:17:40.749 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:40.788 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:41.032 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T11:17:41.060 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T11:17:41.099 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T11:17:41.176 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-25T11:17:41.203 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:41.242 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:41.512 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package pkg-config. 2024-04-25T11:17:41.528 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-colorama. 2024-04-25T11:17:41.540 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T11:17:41.556 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-25T11:17:41.579 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T11:17:41.586 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-25T11:17:41.840 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T11:17:41.847 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephfs-shell. 2024-04-25T11:17:41.856 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T11:17:41.863 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:41.898 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T11:17:41.906 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:42.166 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephfs-top. 2024-04-25T11:17:42.182 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T11:17:42.225 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:42.310 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T11:17:42.338 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T11:17:42.376 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T11:17:42.502 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T11:17:42.518 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:42.560 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:42.679 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T11:17:42.707 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T11:17:42.746 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T11:17:42.847 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T11:17:42.874 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T11:17:42.913 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T11:17:43.083 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T11:17:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T11:17:43.149 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T11:17:43.251 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T11:17:43.278 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T11:17:43.317 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T11:17:43.427 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T11:17:43.455 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T11:17:43.494 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T11:17:43.655 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T11:17:43.682 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T11:17:43.721 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T11:17:43.796 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2024-04-25T11:17:43.813 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-25T11:17:43.846 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T11:17:44.126 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package pkg-config. 2024-04-25T11:17:44.133 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T11:17:44.154 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T11:17:44.161 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T11:17:44.193 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T11:17:44.199 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T11:17:44.479 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T11:17:44.495 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T11:17:44.538 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T11:17:44.671 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T11:17:44.699 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T11:17:44.737 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T11:17:44.982 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T11:17:45.010 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T11:17:45.040 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-toml. 2024-04-25T11:17:45.048 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T11:17:45.068 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-25T11:17:45.106 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T11:17:45.351 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T11:17:45.378 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T11:17:45.409 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T11:17:45.418 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T11:17:45.437 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T11:17:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T11:17:45.755 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T11:17:45.782 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T11:17:45.821 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T11:17:45.821 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T11:17:45.849 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T11:17:45.887 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T11:17:46.083 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T11:17:46.098 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T11:17:46.132 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T11:17:46.159 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T11:17:46.187 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T11:17:46.225 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T11:17:46.436 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-py. 2024-04-25T11:17:46.451 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-25T11:17:46.485 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T11:17:46.772 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T11:17:46.799 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T11:17:46.830 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T11:17:46.962 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2024-04-25T11:17:46.991 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../122-radosgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:47.028 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:47.276 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T11:17:47.303 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T11:17:47.342 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T11:17:47.611 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-toml. 2024-04-25T11:17:47.639 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-25T11:17:47.678 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T11:17:47.989 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T11:17:48.006 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T11:17:48.017 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T11:17:48.034 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:48.055 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T11:17:48.073 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:48.367 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T11:17:48.395 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T11:17:48.434 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T11:17:48.619 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T11:17:48.670 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T11:17:48.697 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T11:17:48.736 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T11:17:48.878 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T11:17:48.979 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T11:17:49.216 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T11:17:49.507 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package radosgw. 2024-04-25T11:17:49.531 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T11:17:49.535 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../122-radosgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:49.573 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:49.963 INFO:teuthology.orchestra.run.smithi136.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T11:17:50.030 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:50.555 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2024-04-25T11:17:50.568 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T11:17:50.584 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T11:17:50.626 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:50.703 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T11:17:50.931 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T11:17:50.979 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T11:17:51.181 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T11:17:51.213 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T11:17:51.299 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T11:17:51.331 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T11:17:51.561 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T11:17:51.597 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T11:17:51.842 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T11:17:51.878 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T11:17:51.969 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T11:17:52.294 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T11:17:52.305 INFO:teuthology.orchestra.run.smithi157.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T11:17:52.376 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:52.638 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T11:17:52.889 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-25T11:17:52.911 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user cephadm....done 2024-04-25T11:17:53.059 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T11:17:53.139 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T11:17:53.318 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T11:17:53.399 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T11:17:53.570 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T11:17:53.692 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T11:17:53.696 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T11:17:53.947 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T11:17:53.968 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T11:17:54.206 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T11:17:54.228 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T11:17:54.324 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T11:17:54.470 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:54.626 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T11:17:54.713 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T11:17:54.839 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T11:17:54.948 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-25T11:17:54.978 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T11:17:55.200 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T11:17:55.237 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-25T11:17:55.325 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T11:17:55.489 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T11:17:55.585 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T11:17:55.773 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T11:17:55.964 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T11:17:55.999 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T11:17:56.033 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T11:17:56.054 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T11:17:56.180 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T11:17:56.267 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T11:17:56.527 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T11:17:56.548 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:56.682 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T11:17:56.878 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:17:56.950 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T11:17:57.138 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T11:17:57.209 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T11:17:57.263 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T11:17:57.398 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-25T11:17:57.502 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T11:17:57.603 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T11:17:57.666 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T11:17:57.775 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T11:17:58.018 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T11:17:58.230 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T11:17:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T11:17:58.425 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T11:17:58.473 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T11:17:58.487 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T11:17:58.589 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T11:17:58.607 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T11:17:58.733 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T11:17:58.940 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T11:17:59.026 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T11:17:59.058 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T11:17:59.309 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T11:17:59.402 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T11:17:59.568 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T11:17:59.636 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T11:17:59.870 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T11:17:59.995 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T11:17:59.996 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T11:18:00.289 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-25T11:18:00.598 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T11:18:00.673 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T11:18:00.915 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T11:18:00.958 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T11:18:01.016 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T11:18:01.117 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T11:18:01.209 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T11:18:01.335 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:01.418 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T11:18:01.636 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T11:18:01.828 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T11:18:01.954 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T11:18:02.070 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T11:18:02.448 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T11:18:02.749 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-25T11:18:03.033 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T11:18:03.168 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T11:18:03.368 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T11:18:03.419 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:03.553 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T11:18:03.618 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T11:18:03.761 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:03.813 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T11:18:03.939 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T11:18:04.054 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T11:18:04.073 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T11:18:04.358 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T11:18:04.381 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T11:18:04.617 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T11:18:04.868 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:04.994 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T11:18:05.287 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T11:18:05.538 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T11:18:05.578 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T11:18:05.789 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T11:18:05.804 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:05.922 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T11:18:06.049 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T11:18:06.182 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T11:18:06.300 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T11:18:06.308 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T11:18:06.434 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T11:18:06.559 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T11:18:06.668 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T11:18:06.735 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T11:18:06.936 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T11:18:06.986 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T11:18:07.062 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T11:18:07.237 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:07.346 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T11:18:07.347 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T11:18:07.640 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T11:18:07.648 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T11:18:07.882 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T11:18:07.907 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T11:18:08.033 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T11:18:08.125 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T11:18:08.184 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T11:18:08.272 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T11:18:08.368 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T11:18:08.510 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T11:18:08.611 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T11:18:08.618 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T11:18:08.895 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T11:18:09.021 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T11:18:09.146 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T11:18:09.314 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T11:18:09.406 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T11:18:09.415 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T11:18:09.532 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T11:18:09.699 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T11:18:09.816 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:09.942 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T11:18:10.001 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T11:18:10.259 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T11:18:10.260 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T11:18:10.386 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T11:18:10.512 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T11:18:10.543 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T11:18:10.601 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T11:18:10.811 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T11:18:10.856 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T11:18:10.957 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T11:18:11.014 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T11:18:11.274 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T11:18:11.325 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T11:18:11.550 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T11:18:11.575 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T11:18:11.702 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T11:18:11.818 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T11:18:11.969 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T11:18:12.095 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T11:18:12.195 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T11:18:12.422 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T11:18:12.462 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T11:18:12.706 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T11:18:12.789 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T11:18:12.916 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:12.966 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T11:18:13.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:13.168 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T11:18:13.169 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T11:18:13.437 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T11:18:13.503 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:13.612 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:13.705 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T11:18:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T11:18:13.956 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T11:18:13.982 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:14.108 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:14.242 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:14.316 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T11:18:14.472 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T11:18:14.472 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T11:18:14.609 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T11:18:14.945 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T11:18:14.968 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.069 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.071 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.186 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.197 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.298 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T11:18:15.337 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.472 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T11:18:15.626 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:15.752 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.253 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T11:18:16.530 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.673 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.675 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.774 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.801 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.927 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:16.978 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T11:18:16.979 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T11:18:17.061 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:17.179 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:17.288 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:17.550 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:17.677 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:17.789 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2024-04-25T11:18:18.140 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2024-04-25T11:18:18.162 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:18.288 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:18.308 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2024-04-25T11:18:18.324 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T11:18:18.406 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T11:18:18.462 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T11:18:18.769 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T11:18:19.264 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:19.480 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:18:19.481 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:18:19.593 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:19.718 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:19.845 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:19.971 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:20.023 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:20.105 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:20.149 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:20.223 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:20.619 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:18:20.620 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:18:20.707 INFO:teuthology.orchestra.run.smithi157.stdout:Adding group ceph....done 2024-04-25T11:18:21.067 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:21.100 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user ceph....done 2024-04-25T11:18:21.319 INFO:teuthology.orchestra.run.smithi157.stdout:Setting system user ceph properties....done 2024-04-25T11:18:21.334 INFO:teuthology.orchestra.run.smithi157.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T11:18:21.379 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T11:18:21.471 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T11:18:21.786 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T11:18:21.819 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:22.035 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T11:18:22.035 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T11:18:22.366 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:22.513 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:22.634 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:18:22.634 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T11:18:22.793 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T11:18:22.793 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T11:18:23.117 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:23.243 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:23.296 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:23.646 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T11:18:23.646 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T11:18:23.705 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:18:23.705 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T11:18:24.103 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:24.149 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:24.292 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:24.434 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:24.448 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T11:18:24.691 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T11:18:24.691 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T11:18:24.930 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.143 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T11:18:25.143 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T11:18:25.169 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.312 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.421 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.556 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.615 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:25.826 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T11:18:25.826 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T11:18:25.909 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T11:18:26.042 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T11:18:26.195 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T11:18:26.317 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T11:18:26.587 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T11:18:26.877 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T11:18:27.060 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:27.186 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:27.346 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:27.493 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T11:18:27.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-25T11:18:27.540 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T11:18:27.540 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T11:18:27.554 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-04-25T11:18:27.754 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-04-25T11:18:27.755 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:18:27.946 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:18:27.947 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:18:27.947 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:18:27.947 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:18:27.948 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-04-25T11:18:27.948 INFO:teuthology.orchestra.run.smithi136.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-25T11:18:27.948 INFO:teuthology.orchestra.run.smithi136.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-25T11:18:27.948 INFO:teuthology.orchestra.run.smithi136.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-25T11:18:27.948 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-25T11:18:27.949 INFO:teuthology.orchestra.run.smithi136.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T11:18:27.949 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-04-25T11:18:27.950 INFO:teuthology.orchestra.run.smithi136.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-25T11:18:27.950 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-25T11:18:27.950 INFO:teuthology.orchestra.run.smithi136.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-25T11:18:27.991 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-25T11:18:27.992 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-25T11:18:27.992 INFO:teuthology.orchestra.run.smithi136.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T11:18:27.992 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2024-04-25T11:18:27.993 INFO:teuthology.orchestra.run.smithi136.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-25T11:18:28.005 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:28.070 INFO:teuthology.orchestra.run.smithi136.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-25T11:18:28.070 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 55.7 MB of archives. 2024-04-25T11:18:28.070 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-25T11:18:28.070 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-25T11:18:28.157 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:28.285 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:28.322 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-25T11:18:28.332 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-25T11:18:28.403 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-25T11:18:28.404 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-25T11:18:28.406 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-25T11:18:28.424 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-25T11:18:28.425 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-25T11:18:28.428 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-25T11:18:28.431 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-25T11:18:28.431 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-25T11:18:28.433 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-25T11:18:28.442 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T11:18:28.446 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-25T11:18:28.464 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-25T11:18:28.465 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-25T11:18:28.466 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-25T11:18:28.795 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T11:18:28.929 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T11:18:29.073 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T11:18:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-25T11:18:29.090 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-25T11:18:29.091 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-25T11:18:29.095 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-25T11:18:29.096 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-25T11:18:29.104 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-25T11:18:29.106 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-25T11:18:29.119 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-25T11:18:29.119 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-25T11:18:29.152 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-25T11:18:29.174 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-25T11:18:29.205 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-25T11:18:29.317 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-25T11:18:29.323 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-25T11:18:29.324 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-25T11:18:29.325 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-25T11:18:29.326 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-25T11:18:29.343 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-25T11:18:29.345 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-25T11:18:29.666 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-25T11:18:29.736 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-25T11:18:29.736 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-25T11:18:29.737 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-25T11:18:29.737 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-25T11:18:29.737 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-25T11:18:29.756 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-25T11:18:29.757 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-25T11:18:30.163 INFO:teuthology.orchestra.run.smithi136.stdout:Preconfiguring packages ... 2024-04-25T11:18:30.182 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T11:18:30.323 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 55.7 MB in 2s (31.7 MB/s) 2024-04-25T11:18:30.639 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T11:18:30.645 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:30.808 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:30.828 INFO:teuthology.orchestra.run.smithi157.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T11:18:30.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-25T11:18:30.929 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-04-25T11:18:31.116 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-04-25T11:18:31.116 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-04-25T11:18:31.290 INFO:teuthology.orchestra.run.smithi157.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-25T11:18:31.290 INFO:teuthology.orchestra.run.smithi157.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-25T11:18:31.290 INFO:teuthology.orchestra.run.smithi157.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-25T11:18:31.290 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T11:18:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T11:18:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T11:18:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T11:18:31.291 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-25T11:18:31.292 INFO:teuthology.orchestra.run.smithi157.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-25T11:18:31.293 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-25T11:18:31.334 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-04-25T11:18:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-25T11:18:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-25T11:18:31.335 INFO:teuthology.orchestra.run.smithi157.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-25T11:18:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-25T11:18:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-25T11:18:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-25T11:18:31.336 INFO:teuthology.orchestra.run.smithi157.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T11:18:31.337 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2024-04-25T11:18:31.337 INFO:teuthology.orchestra.run.smithi157.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-25T11:18:31.396 INFO:teuthology.orchestra.run.smithi157.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-25T11:18:31.396 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 55.7 MB of archives. 2024-04-25T11:18:31.396 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-25T11:18:31.396 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-25T11:18:31.428 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-25T11:18:31.571 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-25T11:18:31.584 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-25T11:18:31.628 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-25T11:18:31.629 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-25T11:18:31.630 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-25T11:18:31.641 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-25T11:18:31.642 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-25T11:18:31.644 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-25T11:18:31.646 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-25T11:18:31.646 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-25T11:18:31.648 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-25T11:18:31.651 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-25T11:18:31.670 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-25T11:18:31.670 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-25T11:18:31.672 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-25T11:18:32.099 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-25T11:18:32.102 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-25T11:18:32.102 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-25T11:18:32.105 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-25T11:18:32.106 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-25T11:18:32.111 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-25T11:18:32.113 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-25T11:18:32.123 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-25T11:18:32.123 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-25T11:18:32.150 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-25T11:18:32.150 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-25T11:18:32.162 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-25T11:18:32.203 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-25T11:18:32.206 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-25T11:18:32.210 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-25T11:18:32.210 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-25T11:18:32.211 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-25T11:18:32.226 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-25T11:18:32.231 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-25T11:18:32.404 INFO:teuthology.orchestra.run.smithi136.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-25T11:18:32.527 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-25T11:18:32.527 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-25T11:18:32.527 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-25T11:18:32.559 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-25T11:18:32.559 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-25T11:18:32.559 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-25T11:18:32.576 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-25T11:18:32.577 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-25T11:18:32.620 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package flex. 2024-04-25T11:18:32.708 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T11:18:32.714 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:32.752 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-25T11:18:32.983 INFO:teuthology.orchestra.run.smithi157.stdout:Preconfiguring packages ... 2024-04-25T11:18:33.131 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 55.7 MB in 1s (45.2 MB/s) 2024-04-25T11:18:33.276 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T11:18:33.445 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T11:18:33.449 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-25T11:18:33.451 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:33.589 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:33.850 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:18:34.231 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-25T11:18:34.401 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T11:18:34.406 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:34.514 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:35.058 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:35.158 INFO:teuthology.orchestra.run.smithi157.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-25T11:18:35.189 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:35.460 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package flex. 2024-04-25T11:18:35.507 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T11:18:35.519 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T11:18:35.524 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:35.567 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-25T11:18:35.873 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package bison. 2024-04-25T11:18:35.978 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T11:18:35.984 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-25T11:18:36.033 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T11:18:36.286 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T11:18:36.425 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-25T11:18:36.453 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-25T11:18:36.491 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T11:18:36.663 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-25T11:18:36.861 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package dump. 2024-04-25T11:18:36.889 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-25T11:18:36.927 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking dump (0.4b47-3) ... 2024-04-25T11:18:37.217 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package indent. 2024-04-25T11:18:37.233 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:37.275 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T11:18:37.385 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:18:37.536 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-25T11:18:37.553 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-25T11:18:37.594 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T11:18:37.899 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-25T11:18:37.927 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-25T11:18:37.966 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T11:18:37.969 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T11:18:37.974 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:38.101 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:38.218 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-25T11:18:38.234 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-25T11:18:38.277 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T11:18:38.535 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T11:18:38.613 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-25T11:18:38.641 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:38.666 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T11:18:38.689 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T11:18:38.916 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-25T11:18:38.932 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:38.967 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T11:18:39.043 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T11:18:39.219 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjson-perl. 2024-04-25T11:18:39.235 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-25T11:18:39.277 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-25T11:18:39.342 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package bison. 2024-04-25T11:18:39.438 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T11:18:39.443 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-25T11:18:39.485 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T11:18:39.573 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-25T11:18:39.600 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-25T11:18:39.656 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T11:18:40.020 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-25T11:18:40.047 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-25T11:18:40.086 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T11:18:40.389 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package dump. 2024-04-25T11:18:40.417 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-25T11:18:40.447 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking dump (0.4b47-3) ... 2024-04-25T11:18:40.726 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package indent. 2024-04-25T11:18:40.743 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:40.785 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T11:18:41.054 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-25T11:18:41.070 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-25T11:18:41.112 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T11:18:41.326 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-25T11:18:41.354 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T11:18:41.392 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:18:41.418 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-25T11:18:41.446 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-25T11:18:41.485 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T11:18:41.654 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-25T11:18:41.681 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T11:18:41.720 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:18:41.821 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-25T11:18:41.838 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-25T11:18:41.889 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T11:18:42.040 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-25T11:18:42.068 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T11:18:42.106 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:18:42.317 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-25T11:18:42.344 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:42.392 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T11:18:42.460 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-25T11:18:42.488 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-25T11:18:42.526 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T11:18:42.678 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-25T11:18:42.705 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-25T11:18:42.761 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T11:18:42.846 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-25T11:18:42.874 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T11:18:42.913 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T11:18:43.014 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjson-perl. 2024-04-25T11:18:43.041 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-25T11:18:43.080 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-25T11:18:43.249 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-25T11:18:43.277 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:43.318 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:43.434 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-25T11:18:43.461 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-25T11:18:43.508 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T11:18:43.644 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-25T11:18:43.672 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:43.710 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:18:44.021 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-25T11:18:44.046 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:44.088 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:18:44.316 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-25T11:18:44.344 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T11:18:44.382 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:18:44.912 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-25T11:18:44.939 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-25T11:18:44.978 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T11:18:45.253 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-25T11:18:45.281 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T11:18:45.328 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:18:45.383 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package modemmanager. 2024-04-25T11:18:45.411 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T11:18:45.449 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:18:45.614 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-25T11:18:45.630 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T11:18:45.672 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:18:46.000 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-25T11:18:46.028 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T11:18:46.075 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:18:46.089 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package network-manager. 2024-04-25T11:18:46.117 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T11:18:46.165 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-25T11:18:46.495 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-25T11:18:46.522 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-25T11:18:46.569 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T11:18:46.730 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ppp. 2024-04-25T11:18:46.747 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-25T11:18:47.012 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T11:18:47.341 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-25T11:18:47.344 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package pptp-linux. 2024-04-25T11:18:47.368 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T11:18:47.372 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-25T11:18:47.399 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T11:18:47.402 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-25T11:18:47.672 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-25T11:18:47.700 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-25T11:18:47.738 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-25T11:18:47.743 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-25T11:18:47.772 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:47.810 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:48.050 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-25T11:18:48.078 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-25T11:18:48.116 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client-common (238) ... 2024-04-25T11:18:48.171 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-25T11:18:48.199 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:48.238 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:18:48.412 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-25T11:18:48.440 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:48.478 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:48.549 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-25T11:18:48.577 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:48.607 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:18:48.843 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-25T11:18:48.871 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T11:18:48.910 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:18:49.042 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-common. 2024-04-25T11:18:49.070 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-25T11:18:49.127 INFO:teuthology.orchestra.run.smithi136.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-25T11:18:49.156 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-common (238) ... 2024-04-25T11:18:49.414 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-25T11:18:49.441 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-25T11:18:49.480 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T11:18:49.608 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-14. 2024-04-25T11:18:49.636 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:49.678 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:49.783 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package modemmanager. 2024-04-25T11:18:49.811 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T11:18:49.850 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:18:50.490 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package network-manager. 2024-04-25T11:18:50.518 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T11:18:50.573 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-25T11:18:51.148 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ppp. 2024-04-25T11:18:51.175 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-25T11:18:51.230 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql. 2024-04-25T11:18:51.259 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-25T11:18:51.296 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql (14+238) ... 2024-04-25T11:18:51.451 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T11:18:51.523 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client. 2024-04-25T11:18:51.540 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-25T11:18:51.582 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client (14+238) ... 2024-04-25T11:18:51.769 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package pptp-linux. 2024-04-25T11:18:51.797 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-25T11:18:51.826 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-25T11:18:51.835 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-25T11:18:51.851 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-25T11:18:51.892 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-25T11:18:52.088 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-25T11:18:52.105 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-25T11:18:52.172 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-25T11:18:52.212 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package tcl8.6. 2024-04-25T11:18:52.240 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-25T11:18:52.278 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T11:18:52.548 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package tcl. 2024-04-25T11:18:52.564 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-25T11:18:52.602 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-25T11:18:52.611 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-25T11:18:52.631 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-25T11:18:52.685 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-client-common (238) ... 2024-04-25T11:18:52.943 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package wpasupplicant. 2024-04-25T11:18:52.971 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-25T11:18:53.006 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-25T11:18:53.009 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T11:18:53.033 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:53.072 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:53.347 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-25T11:18:53.375 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-25T11:18:53.413 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-25T11:18:53.594 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-common. 2024-04-25T11:18:53.622 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-25T11:18:53.671 INFO:teuthology.orchestra.run.smithi157.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-25T11:18:53.692 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-common (238) ... 2024-04-25T11:18:53.792 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-25T11:18:53.820 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-25T11:18:53.867 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T11:18:54.094 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-14. 2024-04-25T11:18:54.171 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client-common (238) ... 2024-04-25T11:18:54.171 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T11:18:54.171 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:54.357 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-25T11:18:54.483 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T11:18:54.609 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:54.743 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T11:18:54.852 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T11:18:54.987 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:55.532 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-25T11:18:55.579 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql. 2024-04-25T11:18:55.606 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-25T11:18:55.618 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T11:18:55.637 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql (14+238) ... 2024-04-25T11:18:55.881 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-client. 2024-04-25T11:18:55.898 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-25T11:18:55.939 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-client (14+238) ... 2024-04-25T11:18:55.981 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T11:18:55.981 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T11:18:56.192 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-25T11:18:56.208 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-25T11:18:56.250 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-25T11:18:56.463 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T11:18:56.553 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package tcl8.6. 2024-04-25T11:18:56.581 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-25T11:18:56.597 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T11:18:56.619 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T11:18:56.723 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:18:56.847 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package tcl. 2024-04-25T11:18:56.849 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-25T11:18:56.875 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-25T11:18:56.918 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-25T11:18:56.975 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T11:18:57.101 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T11:18:57.227 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T11:18:57.267 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package wpasupplicant. 2024-04-25T11:18:57.295 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-25T11:18:57.333 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T11:18:57.345 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T11:18:57.712 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-25T11:18:57.741 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-25T11:18:57.779 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-25T11:18:57.967 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T11:18:58.090 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-25T11:18:58.093 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up dump (0.4b47-3) ... 2024-04-25T11:18:58.107 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-25T11:18:58.148 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T11:18:58.203 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-25T11:18:58.295 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:18:58.404 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T11:18:58.446 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-client-common (238) ... 2024-04-25T11:18:58.522 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-25T11:18:58.648 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-25T11:18:58.654 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-25T11:18:58.774 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T11:18:58.780 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T11:18:58.866 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-25T11:18:58.923 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:58.934 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T11:18:59.033 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T11:18:59.060 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T11:18:59.134 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T11:18:59.237 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T11:18:59.252 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:18:59.363 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:18:59.498 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-25T11:18:59.665 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client (14+238) ... 2024-04-25T11:18:59.773 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-25T11:18:59.791 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-25T11:18:59.882 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout:which means they are currently configured by ifupdown: 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout:- enp3s0f1 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout:remove their configuration from /etc/network/interfaces. 2024-04-25T11:19:00.209 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:00.224 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T11:19:00.224 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T11:19:00.333 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-25T11:19:00.659 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-25T11:19:00.702 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T11:19:00.828 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T11:19:00.954 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T11:19:01.080 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-25T11:19:01.206 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T11:19:01.227 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-25T11:19:01.334 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T11:19:01.450 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T11:19:01.551 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T11:19:01.727 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:19:01.896 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T11:19:02.030 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-25T11:19:02.158 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-common (238) ... 2024-04-25T11:19:02.230 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T11:19:02.357 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up dump (0.4b47-3) ... 2024-04-25T11:19:02.570 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-25T11:19:02.717 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:19:03.038 INFO:teuthology.orchestra.run.smithi136.stdout:Adding user postgres to group ssl-cert 2024-04-25T11:19:03.152 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T11:19:03.262 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:03.262 INFO:teuthology.orchestra.run.smithi136.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-25T11:19:03.277 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-25T11:19:03.404 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-25T11:19:03.445 INFO:teuthology.orchestra.run.smithi136.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-25T11:19:03.445 INFO:teuthology.orchestra.run.smithi136.stdout:Removing obsolete dictionary files: 2024-04-25T11:19:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T11:19:03.620 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-25T11:19:03.698 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T11:19:03.824 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T11:19:03.992 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T11:19:04.006 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-25T11:19:04.093 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:19:04.219 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-25T11:19:04.383 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:19:04.387 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-client (14+238) ... 2024-04-25T11:19:04.513 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-25T11:19:04.692 INFO:teuthology.orchestra.run.smithi136.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-25T11:19:04.705 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-25T11:19:04.757 INFO:teuthology.orchestra.run.smithi136.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:This user must also own the server process. 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:The database cluster will be initialized with locale "en_US". 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:The default text search configuration will be set to "english". 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:Data page checksums are disabled. 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:creating subdirectories ... ok 2024-04-25T11:19:04.758 INFO:teuthology.orchestra.run.smithi136.stdout:selecting dynamic shared memory implementation ... posix 2024-04-25T11:19:04.773 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default max_connections ... 100 2024-04-25T11:19:04.792 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default shared_buffers ... 128MB 2024-04-25T11:19:04.802 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default time zone ... Etc/UTC 2024-04-25T11:19:04.803 INFO:teuthology.orchestra.run.smithi136.stdout:creating configuration files ... ok 2024-04-25T11:19:04.918 INFO:teuthology.orchestra.run.smithi136.stdout:running bootstrap script ... ok 2024-04-25T11:19:04.966 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:04.966 INFO:teuthology.orchestra.run.smithi157.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-25T11:19:04.966 INFO:teuthology.orchestra.run.smithi157.stdout:which means they are currently configured by ifupdown: 2024-04-25T11:19:04.966 INFO:teuthology.orchestra.run.smithi157.stdout:- enp3s0f1 2024-04-25T11:19:04.966 INFO:teuthology.orchestra.run.smithi157.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-25T11:19:04.967 INFO:teuthology.orchestra.run.smithi157.stdout:remove their configuration from /etc/network/interfaces. 2024-04-25T11:19:04.967 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:05.077 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-25T11:19:05.377 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-25T11:19:05.570 INFO:teuthology.orchestra.run.smithi136.stdout:performing post-bootstrap initialization ... ok 2024-04-25T11:19:05.694 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-25T11:19:06.199 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T11:19:06.250 INFO:teuthology.orchestra.run.smithi136.stdout:syncing data to disk ... ok 2024-04-25T11:19:06.325 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T11:19:06.452 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-25T11:19:06.577 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-common (238) ... 2024-04-25T11:19:06.637 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-25T11:19:07.474 INFO:teuthology.orchestra.run.smithi157.stdout:Adding user postgres to group ssl-cert 2024-04-25T11:19:07.730 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:07.731 INFO:teuthology.orchestra.run.smithi157.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-25T11:19:07.922 INFO:teuthology.orchestra.run.smithi157.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-25T11:19:07.922 INFO:teuthology.orchestra.run.smithi157.stdout:Removing obsolete dictionary files: 2024-04-25T11:19:08.464 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-25T11:19:08.879 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T11:19:09.206 INFO:teuthology.orchestra.run.smithi157.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-25T11:19:09.230 INFO:teuthology.orchestra.run.smithi157.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:This user must also own the server process. 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:The database cluster will be initialized with locale "en_US". 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:The default text search configuration will be set to "english". 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:09.279 INFO:teuthology.orchestra.run.smithi157.stdout:Data page checksums are disabled. 2024-04-25T11:19:09.280 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:09.280 INFO:teuthology.orchestra.run.smithi157.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-25T11:19:09.280 INFO:teuthology.orchestra.run.smithi157.stdout:creating subdirectories ... ok 2024-04-25T11:19:09.280 INFO:teuthology.orchestra.run.smithi157.stdout:selecting dynamic shared memory implementation ... posix 2024-04-25T11:19:09.292 INFO:teuthology.orchestra.run.smithi157.stdout:selecting default max_connections ... 100 2024-04-25T11:19:09.309 INFO:teuthology.orchestra.run.smithi157.stdout:selecting default shared_buffers ... 128MB 2024-04-25T11:19:09.318 INFO:teuthology.orchestra.run.smithi157.stdout:selecting default time zone ... Etc/UTC 2024-04-25T11:19:09.319 INFO:teuthology.orchestra.run.smithi157.stdout:creating configuration files ... ok 2024-04-25T11:19:09.390 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:19:09.431 INFO:teuthology.orchestra.run.smithi157.stdout:running bootstrap script ... ok 2024-04-25T11:19:09.516 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-25T11:19:09.642 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql (14+238) ... 2024-04-25T11:19:09.768 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:19:09.877 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:19:10.110 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T11:19:10.110 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T11:19:10.161 INFO:teuthology.orchestra.run.smithi157.stdout:performing post-bootstrap initialization ... ok 2024-04-25T11:19:10.663 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T11:19:10.747 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-25T11:19:10.837 INFO:teuthology.orchestra.run.smithi157.stdout:syncing data to disk ... ok 2024-04-25T11:19:11.058 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T11:19:11.210 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T11:19:11.218 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-25T11:19:12.505 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-25T11:19:12.638 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T11:19:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T11:19:13.128 DEBUG:teuthology.parallel:result is None 2024-04-25T11:19:13.986 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:19:14.111 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-25T11:19:14.229 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up postgresql (14+238) ... 2024-04-25T11:19:14.346 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T11:19:14.472 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T11:19:14.731 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T11:19:14.732 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T11:19:15.267 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T11:19:15.368 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-25T11:19:15.678 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T11:19:15.805 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T11:19:17.133 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-25T11:19:17.242 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T11:19:17.761 INFO:teuthology.orchestra.run.smithi157.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T11:19:17.766 DEBUG:teuthology.parallel:result is None 2024-04-25T11:19:17.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:19:17.994 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2024-04-25T11:19:18.013 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:19:18.014 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:19:18.014 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291-g994e2f20-1jammy is installed. 2024-04-25T11:19:18.015 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:19:18.159 DEBUG:teuthology.orchestra.run.smithi157:> dpkg-query -W -f '${Version}' ceph 2024-04-25T11:19:18.178 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:19:18.178 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T11:19:18.178 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291-g994e2f20-1jammy is installed. 2024-04-25T11:19:18.181 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T11:19:18.181 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:18.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T11:19:18.195 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:18.195 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T11:19:18.234 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T11:19:18.234 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:18.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T11:19:18.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T11:19:18.307 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:18.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T11:19:18.320 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T11:19:18.373 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T11:19:18.373 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:18.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T11:19:18.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T11:19:18.439 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:18.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T11:19:18.451 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T11:19:18.510 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T11:19:18.510 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:18.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T11:19:18.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T11:19:18.579 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:18.579 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T11:19:18.592 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T11:19:18.645 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T11:19:18.732 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T11:19:18.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-04-25T11:19:18.735 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-04-25T11:19:18.748 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T11:19:18.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T11:19:18.789 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T11:19:18.804 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T11:19:18.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T11:19:18.845 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T11:19:18.866 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T11:19:18.866 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T11:19:18.866 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T11:19:18.866 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:18.866 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T11:19:18.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T11:19:18.983 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:18.983 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T11:19:18.989 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T11:19:19.072 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T11:19:19.073 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T11:19:19.073 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673123', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhMlJn51XAwCMqALUCDGHwKhofHRhHNVUyws1BBBbvddkzmyoJ8pvvTWO0QJINIKuVttVKdlB1vfVTsFPRx53w=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4+u+4Hm0cZbDUoP3PAPqtfxjIWHTQkiFhAIU6LidCBYLCwNpcF8ElO7HNne+NYRSUXDP1giHO1/eP5PU4RKoA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T11:19:19.073 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T11:19:19.076 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T11:19:19.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T11:19:19.125 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T11:19:19.171 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:19.171 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T11:19:19.184 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T11:19:19.184 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 879 Links: 1 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-25 11:12:57.341552619 +0000 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-25 11:12:57.077554974 +0000 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-25 11:12:57.077554974 +0000 2024-04-25T11:19:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-25T11:19:19.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T11:19:19.287 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-25T11:19:19.287 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-25T11:19:19.287 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00045809 s, 1.1 MB/s 2024-04-25T11:19:19.288 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T11:19:19.338 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-25 11:12:57.841548159 +0000 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-25 11:12:57.561550657 +0000 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-25 11:12:57.561550657 +0000 2024-04-25T11:19:19.384 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-25T11:19:19.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T11:19:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-25T11:19:19.439 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-25T11:19:19.439 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000358308 s, 1.4 MB/s 2024-04-25T11:19:19.440 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T11:19:19.489 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-25 11:12:58.341543698 +0000 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-25 11:12:58.061546196 +0000 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-25 11:12:58.061546196 +0000 2024-04-25T11:19:19.536 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-25T11:19:19.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T11:19:19.590 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-25T11:19:19.590 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-25T11:19:19.590 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000340507 s, 1.5 MB/s 2024-04-25T11:19:19.591 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T11:19:19.641 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-04-25T11:19:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T11:19:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:19.688 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-04-25T11:19:19.688 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:19.689 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-25 11:12:58.837539273 +0000 2024-04-25T11:19:19.689 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-25 11:12:58.589541486 +0000 2024-04-25T11:19:19.689 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-25 11:12:58.589541486 +0000 2024-04-25T11:19:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-25T11:19:19.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T11:19:19.742 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-25T11:19:19.743 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-25T11:19:19.743 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000422739 s, 1.2 MB/s 2024-04-25T11:19:19.744 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T11:19:19.793 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:19.793 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:19.794 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T11:19:19.799 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T11:19:19.799 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-04-25T11:19:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T11:19:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:19.846 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-25T11:19:19.846 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:19.847 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-04-25 11:12:57.291399764 +0000 2024-04-25T11:19:19.847 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-04-25 11:12:57.075401685 +0000 2024-04-25T11:19:19.847 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-04-25 11:12:57.075401685 +0000 2024-04-25T11:19:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-04-25T11:19:19.847 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T11:19:19.901 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-04-25T11:19:19.901 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-04-25T11:19:19.902 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000291284 s, 1.8 MB/s 2024-04-25T11:19:19.903 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T11:19:19.955 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-04-25 11:12:57.511397808 +0000 2024-04-25T11:19:20.007 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-04-25 11:12:57.507397845 +0000 2024-04-25T11:19:20.008 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-04-25 11:12:57.507397845 +0000 2024-04-25T11:19:20.008 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-04-25T11:19:20.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T11:19:20.063 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-04-25T11:19:20.063 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-04-25T11:19:20.064 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000207589 s, 2.5 MB/s 2024-04-25T11:19:20.066 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T11:19:20.116 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-04-25 11:12:58.279390982 +0000 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-04-25 11:12:58.007393399 +0000 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-04-25 11:12:58.007393399 +0000 2024-04-25T11:19:20.162 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-04-25T11:19:20.163 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T11:19:20.216 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-04-25T11:19:20.216 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-04-25T11:19:20.216 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000312132 s, 1.6 MB/s 2024-04-25T11:19:20.217 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T11:19:20.268 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-04-25 11:12:58.755386752 +0000 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-04-25 11:12:58.495389063 +0000 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-04-25 11:12:58.495389063 +0000 2024-04-25T11:19:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2024-04-25T11:19:20.314 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T11:19:20.369 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-04-25T11:19:20.369 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-04-25T11:19:20.369 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000190147 s, 2.7 MB/s 2024-04-25T11:19:20.370 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T11:19:20.419 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:20.419 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T11:19:20.419 INFO:tasks.ceph:Generating config... 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T11:19:20.421 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-25T11:19:20.421 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T11:19:20.422 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T11:19:20.423 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T11:19:20.423 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T11:19:20.423 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T11:19:20.423 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T11:19:20.423 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T11:19:20.424 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T11:19:20.424 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T11:19:20.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T11:19:20.489 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T11:19:20.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T11:19:20.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T11:19:20.578 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.136'), ('mon.b', '172.21.15.157'), ('mon.c', '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]')] 2024-04-25T11:19:20.579 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.136,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T11:19:20.580 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:20.580 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T11:19:20.623 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.136 --add b 172.21.15.157 --addv c '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:20.734 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:20.734 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: generated fsid 6dead98d-43fa-4435-a3fe-cc7c15301a50 2024-04-25T11:19:20.734 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = quincy 2024-04-25T11:19:20.734 INFO:teuthology.orchestra.run.smithi136.stdout:epoch 0 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:fsid 6dead98d-43fa-4435-a3fe-cc7c15301a50 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:last_changed 2024-04-25T11:19:20.740527+0000 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:created 2024-04-25T11:19:20.740527+0000 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:min_mon_release 17 (quincy) 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:election_strategy: 1 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:2: [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] mon.c 2024-04-25T11:19:20.735 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T11:19:20.739 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T11:19:20.745 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6dead98d-43fa-4435-a3fe-cc7c15301a50... 2024-04-25T11:19:20.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T11:19:20.789 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T11:19:20.816 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-04-25T11:19:20.816 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-04-25T11:19:20.816 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-04-25T11:19:20.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T11:19:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "172.21.15.136,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 6dead98d-43fa-4435-a3fe-cc7c15301a50 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-04-25T11:19:20.819 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-04-25T11:19:20.820 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-04-25T11:19:20.821 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 20 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 20 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: filestore flush min = 0 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-04-25T11:19:20.822 INFO:teuthology.orchestra.run.smithi136.stdout: osd blocked scrub grace period = 3600 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd max markdown count = 1000 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: osd op complaint time = 180 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-04-25T11:19:20.823 INFO:teuthology.orchestra.run.smithi136.stdout: debug client = 20 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T11:19:20.824 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon down mkfs grace = 300 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: mon op complaint time = 120 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-04-25T11:19:20.825 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: client mount timeout = 600 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: client_shutdown_timeout = 120 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug client = 20 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: rados mon op timeout = 900 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout: rados osd op timeout = 900 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-04-25T11:19:20.826 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.a] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.c] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.e] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.b] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.d] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.f] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout:[mds] 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug mds = 20 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug mds balancer = 20 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal fragment size max = 10000 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal merge size = 5 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal split bits = 3 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal split size = 100 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds debug frag = True 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds debug scatterstat = True 2024-04-25T11:19:20.827 INFO:teuthology.orchestra.run.smithi136.stdout: mds oft prefetch dirfrags = True 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: mds op complaint time = 180 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: mds verify scatter = True 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: mds_dir_prefetch = True 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: mds_sleep_rank_change = 5000000.0 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: osd op complaint time = 180 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: rados mon op timeout = 900 2024-04-25T11:19:20.828 INFO:teuthology.orchestra.run.smithi136.stdout: rados osd op timeout = 900 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.829 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-04-25T11:19:20.830 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-04-25T11:19:20.831 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.136,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 6dead98d-43fa-4435-a3fe-cc7c15301a50 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-04-25T11:19:20.832 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.833 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-04-25T11:19:20.834 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: filestore flush min = 0 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: osd blocked scrub grace period = 3600 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-04-25T11:19:20.835 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug client = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-04-25T11:19:20.836 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.837 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: mon down mkfs grace = 300 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: mon op complaint time = 120 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: client mount timeout = 600 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: client_shutdown_timeout = 120 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: debug client = 20 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-04-25T11:19:20.838 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.e] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.f] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout:[mds] 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds = 20 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds balancer = 20 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: mds bal fragment size max = 10000 2024-04-25T11:19:20.839 INFO:teuthology.orchestra.run.smithi157.stdout: mds bal merge size = 5 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds bal split bits = 3 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds bal split size = 100 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug frag = True 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug scatterstat = True 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds oft prefetch dirfrags = True 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds op complaint time = 180 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds verify scatter = True 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds_dir_prefetch = True 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: mds_sleep_rank_change = 5000000.0 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-04-25T11:19:20.840 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-04-25T11:19:20.842 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T11:19:20.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T11:19:20.962 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T11:19:20.963 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:20.963 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T11:19:20.968 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:20.968 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T11:19:21.017 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:19:21.017 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:21.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T11:19:21.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T11:19:21.081 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:21.081 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:21.128 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:19:21.128 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:21.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T11:19:21.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T11:19:21.153 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:21.153 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:21.202 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T11:19:21.202 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T11:19:21.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T11:19:21.276 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T11:19:21.283 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T11:19:21.357 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T11:19:21.373 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T11:19:21.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T11:19:21.447 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T11:19:21.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T11:19:21.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T11:19:21.570 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T11:19:21.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T11:19:21.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-25T11:19:21.691 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-25T11:19:21.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-25T11:19:21.707 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T11:19:21.782 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T11:19:21.788 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T11:19:21.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T11:19:21.910 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T11:19:21.917 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T11:19:21.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-25T11:19:22.040 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-25T11:19:22.046 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-25T11:19:22.057 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T11:19:22.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T11:19:22.124 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T11:19:22.139 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T11:19:22.139 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T11:19:22.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T11:19:22.151 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:22.151 INFO:tasks.ceph:role: osd.0 2024-04-25T11:19:22.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:19:22.151 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T11:19:22.214 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:22.215 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:22.713 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-25T11:19:22.716 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-25T11:19:22.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T11:19:22.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T11:19:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:22.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:22.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T11:19:22.885 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:22.885 INFO:tasks.ceph:role: osd.1 2024-04-25T11:19:22.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:19:22.885 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T11:19:22.947 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:22.947 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:22.948 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:23.467 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-25T11:19:23.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-25T11:19:23.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T11:19:23.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T11:19:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:23.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:23.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T11:19:23.645 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:23.645 INFO:tasks.ceph:role: osd.2 2024-04-25T11:19:23.645 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:19:23.645 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:23.708 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:24.199 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-25T11:19:24.201 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-25T11:19:24.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T11:19:24.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T11:19:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:24.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:24.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T11:19:24.375 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:24.375 INFO:tasks.ceph:role: osd.3 2024-04-25T11:19:24.375 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:19:24.375 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T11:19:24.440 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:24.440 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:24.441 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:24.950 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-25T11:19:24.952 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-25T11:19:24.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T11:19:25.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T11:19:25.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:25.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:25.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.236+0000 7f4e829767c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T11:19:25.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.240+0000 7f4e829767c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T11:19:25.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.240+0000 7f4e829767c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T11:19:25.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.240+0000 7f4e829767c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T11:19:25.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.240+0000 7f4e829767c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T11:19:25.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:25.244+0000 7f4e829767c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T11:19:28.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T11:19:28.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:28.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T11:19:28.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T11:19:28.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T11:19:28.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T11:19:28.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T11:19:28.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:28.348+0000 7f9c0afdb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T11:19:31.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T11:19:31.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:31.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.452+0000 7ff91f4617c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T11:19:31.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.452+0000 7ff91f4617c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T11:19:31.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.452+0000 7ff91f4617c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T11:19:31.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.452+0000 7ff91f4617c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T11:19:31.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.452+0000 7ff91f4617c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T11:19:31.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:31.456+0000 7ff91f4617c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T11:19:34.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T11:19:34.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:34.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T11:19:34.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T11:19:34.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T11:19:34.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T11:19:34.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T11:19:34.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:34.556+0000 7f3c7ffa07c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T11:19:37.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T11:19:37.549 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T11:19:37.563 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:37.563 INFO:tasks.ceph:role: osd.4 2024-04-25T11:19:37.563 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:19:37.563 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:37.618 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:37.641 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-04-25T11:19:37.643 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-04-25T11:19:37.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T11:19:37.704 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T11:19:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:37.759 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:37.759 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T11:19:37.811 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:37.811 INFO:tasks.ceph:role: osd.5 2024-04-25T11:19:37.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:19:37.811 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:37.865 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:37.888 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-04-25T11:19:37.890 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-04-25T11:19:37.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T11:19:37.949 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T11:19:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:38.004 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T11:19:38.056 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:38.056 INFO:tasks.ceph:role: osd.6 2024-04-25T11:19:38.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:19:38.056 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T11:19:38.109 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:38.109 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:38.109 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:38.110 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:38.133 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-04-25T11:19:38.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-04-25T11:19:38.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T11:19:38.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T11:19:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:38.251 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:38.251 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T11:19:38.305 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T11:19:38.305 INFO:tasks.ceph:role: osd.7 2024-04-25T11:19:38.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:19:38.305 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T11:19:38.360 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T11:19:38.361 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T11:19:38.384 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-04-25T11:19:38.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-04-25T11:19:38.386 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T11:19:38.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T11:19:38.499 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T11:19:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T11:19:38.499 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T11:19:38.650 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:38.653+0000 7fb3448887c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T11:19:41.621 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T11:19:41.634 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:41.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T11:19:41.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T11:19:41.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T11:19:41.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T11:19:41.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T11:19:41.750 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:41.753+0000 7f19880c07c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T11:19:44.714 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T11:19:44.728 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T11:19:44.843 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:44.845+0000 7f3c399ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T11:19:47.829 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T11:19:47.844 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.957+0000 7fd2d20a37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.961+0000 7fd2d20a37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.961+0000 7fd2d20a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.961+0000 7fd2d20a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.961+0000 7fd2d20a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T11:19:47.958 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:47.961+0000 7fd2d20a37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T11:19:49.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:19:49.078 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:19:50.928 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T11:19:50.941 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T11:19:50.942 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:50.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T11:19:50.955 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:50.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T11:19:51.012 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T11:19:51.071 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-25T11:19:51.128 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T11:19:51.184 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T11:19:51.240 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T11:19:51.296 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T11:19:51.351 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.351 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T11:19:51.365 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.365 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T11:19:51.417 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.417 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T11:19:51.469 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.469 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-25T11:19:51.525 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.525 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T11:19:51.577 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.578 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T11:19:51.633 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.633 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T11:19:51.685 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:51.685 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T11:19:51.741 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:51.741 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T11:19:51.747 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T11:19:51.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T11:19:51.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T11:19:51.802 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-04-25T11:19:51.802 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmQzDaEBAACxv8Kf2sx0G5f6pR/PlWrg== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.a] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmejIFGxAA9LSVWFHN4ICj9EYQtCVIgg== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.c] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmllZaIhAAuktENjp+mHanoOg/9NbyZw== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.e] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmv16bKRAAlQQFPWPAN02cxoGFt/jtmg== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA9PCpm0NJCDhAA2ymM8xdn/Mc+QJ7MRp8p7w== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBAPCpmfCP5FBAAjsmDFeW/BCpqlxuWlljGsg== 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-04-25T11:19:51.803 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBDPCpmyCs9GxAAV/79iaWCv/o5DtwYK7YNiw== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBGPCpm7Y5fIRAAzy6UspQvLOgy7OrviBLDRQ== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmDu+jFRAAzDT5H6iwP4FrLNMJwg0/Cg== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.b] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpmvrn2LhAAZ2iBkeqzfZ0untUTxPg98w== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.d] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA5PCpm/8ucNhAAHZPGhi6uNk+pXOxrsgYftw== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.f] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6PCpmmu+6AhAAgEhIxu71m7b7n6/nPlxlCw== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBKPCpm2N4RJxAA8DiESS5cPP0E3lG0ERPn/w== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBNPCpmLtcHLRAAhTsQAacUeqCAEkzL3oE7Dg== 2024-04-25T11:19:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-04-25T11:19:51.805 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBQPCpmw7iTMhAAaT2BjYGK6LBDYRXQtdw4wA== 2024-04-25T11:19:51.805 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-04-25T11:19:51.805 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBTPCpmgwVqORAAuk9lk1hRHThT6Vs3dko8RQ== 2024-04-25T11:19:51.805 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-04-25T11:19:51.805 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6PCpmk2zIBxAAO92HJ9gboT53d2yPG5MWkg== 2024-04-25T11:19:51.806 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-04-25T11:19:51.806 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmQzDaEBAACxv8Kf2sx0G5f6pR/PlWrg== 2024-04-25T11:19:51.806 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-04-25T11:19:51.806 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmejIFGxAA9LSVWFHN4ICj9EYQtCVIgg== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmllZaIhAAuktENjp+mHanoOg/9NbyZw== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.e] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmv16bKRAAlQQFPWPAN02cxoGFt/jtmg== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA9PCpm0NJCDhAA2ymM8xdn/Mc+QJ7MRp8p7w== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBAPCpmfCP5FBAAjsmDFeW/BCpqlxuWlljGsg== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBDPCpmyCs9GxAAV/79iaWCv/o5DtwYK7YNiw== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBGPCpm7Y5fIRAAzy6UspQvLOgy7OrviBLDRQ== 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-04-25T11:19:51.807 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmDu+jFRAAzDT5H6iwP4FrLNMJwg0/Cg== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpmvrn2LhAAZ2iBkeqzfZ0untUTxPg98w== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA5PCpm/8ucNhAAHZPGhi6uNk+pXOxrsgYftw== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.f] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA6PCpmmu+6AhAAgEhIxu71m7b7n6/nPlxlCw== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBKPCpm2N4RJxAA8DiESS5cPP0E3lG0ERPn/w== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBNPCpmLtcHLRAAhTsQAacUeqCAEkzL3oE7Dg== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBQPCpmw7iTMhAAaT2BjYGK6LBDYRXQtdw4wA== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQBTPCpmgwVqORAAuk9lk1hRHThT6Vs3dko8RQ== 2024-04-25T11:19:51.808 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-04-25T11:19:51.809 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQA6PCpmk2zIBxAAO92HJ9gboT53d2yPG5MWkg== 2024-04-25T11:19:51.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T11:19:51.853 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T11:19:51.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:51.935 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.013 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.087 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.161 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.232 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.302 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.376 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T11:19:52.452 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T11:19:52.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.523 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.597 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.672 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T11:19:52.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.747 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.820 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.892 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:52.964 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T11:19:53.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T11:19:53.035 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T11:19:53.108 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T11:19:53.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T11:19:53.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T11:19:53.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T11:19:53.243 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T11:19:53.256 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T11:19:53.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T11:19:53.392 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T11:19:53.444 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T11:19:53.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T11:19:53.580 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:53.583 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T11:19:53.625 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T11:19:53.626 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T11:19:53.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T11:19:53.629 INFO:tasks.ceph.mon.a:Started 2024-04-25T11:19:53.629 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T11:19:53.629 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T11:19:53.671 INFO:tasks.ceph.mon.b:Started 2024-04-25T11:19:53.671 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T11:19:53.671 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T11:19:53.674 INFO:tasks.ceph.mon.c:Started 2024-04-25T11:19:53.674 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T11:19:53.674 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T11:19:53.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T11:19:53.677 INFO:tasks.ceph.mgr.y:Started 2024-04-25T11:19:53.677 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T11:19:53.677 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T11:19:53.679 INFO:tasks.ceph.mgr.x:Started 2024-04-25T11:19:53.679 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T11:19:53.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T11:19:53.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.819+0000 7fa608fd0640 1 Processor -- start 2024-04-25T11:19:53.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.819+0000 7fa608fd0640 1 -- start start 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa608fd0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa608fd0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa608fd0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60415d750 con 0x7fa6040ca290 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa608fd0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa60415da30 con 0x7fa60415ab40 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa608fd0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa60415dd10 con 0x7fa6040b5060 2024-04-25T11:19:53.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56574/0 (socket says 172.21.15.136:56574) 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 learned_addr learned my addr 172.21.15.136:0/744119025 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa601d9b640 1 --1- 172.21.15.136:0/744119025 >> v1:172.21.15.136:6789/0 conn(0x7fa6040ca290 0x7fa60415a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50328/0 (socket says 172.21.15.136:50328) 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 msgr2=0x7fa6040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 msgr2=0x7fa6040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa60259c640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa6040c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa60259c640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa6040c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:19:53.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:53.823+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T11:19:54.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 msgr2=0x7fa6040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 msgr2=0x7fa6040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa60259c640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa6040c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa60259c640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa6040c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T11:19:54.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.027+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T11:19:54.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613817758 0 0) 0x7fa60415d750 con 0x7fa6040ca290 2024-04-25T11:19:54.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5ec003530 con 0x7fa6040ca290 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3299278696 0 0) 0x7fa5ec003530 con 0x7fa6040ca290 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa6040c4520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa6040c4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 msgr2=0x7fa6040ca6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 --2- 172.21.15.136:0/744119025 >> v2:172.21.15.136:3300/0 conn(0x7fa60415ab40 0x7fa6040ca6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60415dff0 con 0x7fa6040ca290 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.075+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fa5f4001680 con 0x7fa6040ca290 2024-04-25T11:19:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 >> v1:172.21.15.136:6789/0 conn(0x7fa6040ca290 legacy=0x7fa60415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5ec0039c0 0x7fa5ec003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa5ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5ec004d50 0x7fa5ec005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa60415da30 con 0x7fa5ec004d50 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa60415dd10 con 0x7fa5ec0039c0 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa5ec00bbc0 con 0x7fa6040b5060 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa60259c640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5ec0039c0 0x7fa5ec003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa602d9d640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5ec004d50 0x7fa5ec005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa5ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5ec0039c0 msgr2=0x7fa5ec003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5ec0039c0 0x7fa5ec003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5ec004d50 msgr2=0x7fa5ec005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:54.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5ec004d50 0x7fa5ec005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.079+0000 7fa601d9b640 1 -- 172.21.15.136:0/744119025 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ec003530 con 0x7fa6040b5060 2024-04-25T11:19:54.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa601d9b640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa5ec004810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4001060 tx=0x7fa5f401cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=447e1e58d5fd5bef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:54.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4007cc0 con 0x7fa6040b5060 2024-04-25T11:19:54.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f40a9020 con 0x7fa6040b5060 2024-04-25T11:19:54.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa60159a640 1 -- 172.21.15.136:0/744119025 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40072f0 con 0x7fa6040b5060 2024-04-25T11:19:54.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 -- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa5ec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 --2- 172.21.15.136:0/744119025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa5ec004810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4001060 tx=0x7fa5f401cb40 comp rx=0 tx=0).stop 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa602d9d640 1 -- 172.21.15.136:0/744119025 reap_dead start 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 -- 172.21.15.136:0/744119025 shutdown_connections 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 -- 172.21.15.136:0/744119025 >> 172.21.15.136:0/744119025 conn(0x7fa604070f70 msgr2=0x7fa604068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 -- 172.21.15.136:0/744119025 shutdown_connections 2024-04-25T11:19:54.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.083+0000 7fa608fd0640 1 -- 172.21.15.136:0/744119025 wait complete. 2024-04-25T11:19:54.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.087+0000 7fa608fd0640 1 Processor -- start 2024-04-25T11:19:54.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.087+0000 7fa608fd0640 1 -- start start 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa604153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa60415c650 0x7fa60414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa60415d3f0 con 0x7fa60414d040 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa604159f10 con 0x7fa60415c650 2024-04-25T11:19:54.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6040c4520 con 0x7fa6040b5060 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa60259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa604153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56612/0 (socket says 172.21.15.136:56612) 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 -- 172.21.15.136:0/2416710114 learned_addr learned my addr 172.21.15.136:0/2416710114 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa601d9b640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa60415c650 0x7fa60414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 -- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 msgr2=0x7fa604153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa604153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 -- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa60415c650 msgr2=0x7fa60414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa60415c650 0x7fa60414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 -- 172.21.15.136:0/2416710114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60414d990 con 0x7fa60414d040 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa602d9d640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6040c9bc0 tx=0x7fa5fc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc003c70 con 0x7fa60414d040 2024-04-25T11:19:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5fc01b070 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa604153c90 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc011a40 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6040b4360 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa5fc004030 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa5fc01f890 con 0x7fa60414d040 2024-04-25T11:19:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.091+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6040cd300 con 0x7fa60414d040 2024-04-25T11:19:54.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.099+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa6040cd300 con 0x7fa60414d040 2024-04-25T11:19:54.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.171+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa6040cea10 con 0x7fa60414d040 2024-04-25T11:19:54.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa5f2ffd640 1 -- 172.21.15.136:0/2416710114 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa6040cea10 con 0x7fa60414d040 2024-04-25T11:19:54.174 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-04-25T11:19:54.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 msgr2=0x7fa60414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6040c9bc0 tx=0x7fa5fc007f60 comp rx=0 tx=0).stop 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 shutdown_connections 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa60414d040 0x7fa60414d450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa60415c650 0x7fa60414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 --2- 172.21.15.136:0/2416710114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6040b5060 0x7fa604153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 >> 172.21.15.136:0/2416710114 conn(0x7fa604070f70 msgr2=0x7fa6040cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 shutdown_connections 2024-04-25T11:19:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:54.175+0000 7fa608fd0640 1 -- 172.21.15.136:0/2416710114 wait complete. 2024-04-25T11:19:54.193 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T11:19:54.193 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T11:19:54.194 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:54.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T11:19:54.205 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:54.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T11:19:54.257 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:54.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T11:19:54.308 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:54.308+0000 7feb9a04c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:19:54.311 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:19:54.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T11:19:54.367 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:54.368 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T11:19:54.381 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:54.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T11:19:54.438 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:54.438 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T11:19:54.492 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-04-25T11:19:54.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T11:19:54.544 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 23f55b63-4108-4987-b1b4-0059929a9aa7 0 2024-04-25T11:19:54.599 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:54.599+0000 7f9cdc3c1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T11:19:54.735 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 Processor -- start 2024-04-25T11:19:54.735 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 -- start start 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 0x7ffb900c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffb900ca290 0x7ffb900ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb9015d750 con 0x7ffb9015a430 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9015da30 con 0x7ffb900ca290 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb987cb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffb9015dd10 con 0x7ffb900b5060 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb96567640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 0x7ffb900c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb95d66640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffb900ca290 0x7ffb900ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb96567640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 0x7ffb900c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47430/0 (socket says 172.21.15.157:47430) 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb96567640 1 -- 172.21.15.157:0/575276503 learned_addr learned my addr 172.21.15.157:0/575276503 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb95d66640 1 -- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 msgr2=0x7ffb900c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb95d66640 1 --2- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 0x7ffb900c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.736+0000 7ffb95d66640 1 -- 172.21.15.157:0/575276503 >> v1:172.21.15.136:6789/0 conn(0x7ffb9015a430 legacy=0x7ffb9015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:54.739 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb95d66640 1 -- 172.21.15.157:0/575276503 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9015dff0 con 0x7ffb900ca290 2024-04-25T11:19:54.740 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb95d66640 1 --2- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900ca290 0x7ffb900ca6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb800097c0 tx=0x7ffb8001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24e97be9a5e6f728 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:54.740 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb95565640 1 -- 172.21.15.157:0/575276503 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb800b5070 con 0x7ffb900ca290 2024-04-25T11:19:54.740 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb95565640 1 -- 172.21.15.157:0/575276503 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb800a8990 con 0x7ffb900ca290 2024-04-25T11:19:54.740 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb95565640 1 -- 172.21.15.157:0/575276503 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb800a8d60 con 0x7ffb900ca290 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 -- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900ca290 msgr2=0x7ffb900ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 --2- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900ca290 0x7ffb900ca6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb800097c0 tx=0x7ffb8001ec90 comp rx=0 tx=0).stop 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 -- 172.21.15.157:0/575276503 shutdown_connections 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 --2- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900ca290 0x7ffb900ca6a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 --2- 172.21.15.157:0/575276503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900b5060 0x7ffb900c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 -- 172.21.15.157:0/575276503 >> 172.21.15.157:0/575276503 conn(0x7ffb90070f70 msgr2=0x7ffb90068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 -- 172.21.15.157:0/575276503 shutdown_connections 2024-04-25T11:19:54.741 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.740+0000 7ffb987cb640 1 -- 172.21.15.157:0/575276503 wait complete. 2024-04-25T11:19:54.742 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 Processor -- start 2024-04-25T11:19:54.743 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- start start 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 0x7ffb900c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb9015a430 0x7ffb900c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9015d3f0 con 0x7ffb9015a430 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb90159f10 con 0x7ffb900b5060 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffb900c4a60 con 0x7ffb900ca290 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55164/0 (socket says 172.21.15.157:55164) 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 -- 172.21.15.157:0/4057318937 learned_addr learned my addr 172.21.15.157:0/4057318937 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb95d66640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 0x7ffb900c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96d68640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb9015a430 0x7ffb900c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.746 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 -- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 msgr2=0x7ffb900c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 0x7ffb900c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 -- 172.21.15.157:0/4057318937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb9015a430 msgr2=0x7ffb900c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb9015a430 0x7ffb900c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 -- 172.21.15.157:0/4057318937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9014f4f0 con 0x7ffb900b5060 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb95d66640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 0x7ffb900c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:54.747 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb96567640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00e820 tx=0x7ffb8c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c017070 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb8c00c940 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb90158b60 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c00cd30 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb90158e20 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffb8c01a070 con 0x7ffb900b5060 2024-04-25T11:19:54.748 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.744+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7ffb8c013470 con 0x7ffb900b5060 2024-04-25T11:19:54.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.748+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb9006a350 con 0x7ffb900b5060 2024-04-25T11:19:54.749 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.748+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffb8c0059a0 con 0x7ffb900b5060 2024-04-25T11:19:54.809 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:54.808+0000 7feb9a04c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:19:54.817 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.816+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "23f55b63-4108-4987-b1b4-0059929a9aa7", "id": 0} v 0) -- 0x7ffb900cadd0 con 0x7ffb900b5060 2024-04-25T11:19:54.824 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.824+0000 7ffb86ffd640 1 -- 172.21.15.157:0/4057318937 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23f55b63-4108-4987-b1b4-0059929a9aa7", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7ffb8c005b80 con 0x7ffb900b5060 2024-04-25T11:19:54.825 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 msgr2=0x7ffb9014efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00e820 tx=0x7ffb8c00ece0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 shutdown_connections 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb9015a430 0x7ffb900c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb900ca290 0x7ffb900c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 --2- 172.21.15.157:0/4057318937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb900b5060 0x7ffb9014efb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 >> 172.21.15.157:0/4057318937 conn(0x7ffb90070f70 msgr2=0x7ffb900cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 shutdown_connections 2024-04-25T11:19:54.826 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.828+0000 7ffb987cb640 1 -- 172.21.15.157:0/4057318937 wait complete. 2024-04-25T11:19:54.845 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 4b178c77-9ab9-43ea-842b-66df2de40110 1 2024-04-25T11:19:54.989 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.988+0000 7fb5797ee640 1 Processor -- start 2024-04-25T11:19:54.989 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.988+0000 7fb5797ee640 1 -- start start 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5797ee640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 0x7fb5740c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5797ee640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5797ee640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5740c3880 con 0x7fb574154060 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5797ee640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb5740c8fd0 con 0x7fb57415a5f0 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5797ee640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb57406d9b0 con 0x7fb57415aa00 2024-04-25T11:19:54.991 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47446/0 (socket says 172.21.15.157:47446) 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 -- 172.21.15.157:0/4131928958 learned_addr learned my addr 172.21.15.157:0/4131928958 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb56bfff640 1 --2- 172.21.15.157:0/4131928958 >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 0x7fb5740c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 -- 172.21.15.157:0/4131928958 >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 msgr2=0x7fb5740c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 --2- 172.21.15.157:0/4131928958 >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 0x7fb5740c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 -- 172.21.15.157:0/4131928958 >> v1:172.21.15.136:6789/0 conn(0x7fb574154060 legacy=0x7fb5740c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 -- 172.21.15.157:0/4131928958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb574154470 con 0x7fb57415aa00 2024-04-25T11:19:54.992 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb56bfff640 1 --2- 172.21.15.157:0/4131928958 >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 0x7fb5740c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:54.993 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5737fe640 1 --2- 172.21.15.157:0/4131928958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb574154640 tx=0x7fb56c01f200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e916ca428a1a3ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:54.993 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5727fc640 1 -- 172.21.15.157:0/4131928958 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0b4070 con 0x7fb57415aa00 2024-04-25T11:19:54.993 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5727fc640 1 -- 172.21.15.157:0/4131928958 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c0af400 con 0x7fb57415aa00 2024-04-25T11:19:54.993 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.992+0000 7fb5727fc640 1 -- 172.21.15.157:0/4131928958 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0af7d0 con 0x7fb57415aa00 2024-04-25T11:19:54.994 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- 172.21.15.157:0/4131928958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 msgr2=0x7fb5740c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 --2- 172.21.15.157:0/4131928958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb574154640 tx=0x7fb56c01f200 comp rx=0 tx=0).stop 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- 172.21.15.157:0/4131928958 shutdown_connections 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 --2- 172.21.15.157:0/4131928958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb5740c76f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 --2- 172.21.15.157:0/4131928958 >> v2:172.21.15.157:3300/0 conn(0x7fb57415a5f0 0x7fb5740c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- 172.21.15.157:0/4131928958 >> 172.21.15.157:0/4131928958 conn(0x7fb5740ca5a0 msgr2=0x7fb5740ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- 172.21.15.157:0/4131928958 shutdown_connections 2024-04-25T11:19:54.995 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- 172.21.15.157:0/4131928958 wait complete. 2024-04-25T11:19:54.997 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 Processor -- start 2024-04-25T11:19:54.997 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:54.996+0000 7fb5797ee640 1 -- start start 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb57415a5f0 0x7fb57414e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb574157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb57415a0d0 con 0x7fb57415a5f0 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb57406df30 con 0x7fb574154060 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb574159d80 con 0x7fb57415aa00 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb572ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb572ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55180/0 (socket says 172.21.15.157:55180) 2024-04-25T11:19:55.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb572ffd640 1 -- 172.21.15.157:0/1232991969 learned_addr learned my addr 172.21.15.157:0/1232991969 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb574157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56bfff640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb57415a5f0 0x7fb57414e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 -- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 msgr2=0x7fb5741532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 -- 172.21.15.157:0/1232991969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb57415a5f0 msgr2=0x7fb57414e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb57415a5f0 0x7fb57414e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 -- 172.21.15.157:0/1232991969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb574153800 con 0x7fb57415aa00 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb572ffd640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5737fe640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb574157b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5740c7c90 tx=0x7fb56c01bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0b4040 con 0x7fb57415aa00 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c0090d0 con 0x7fb57415aa00 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c009510 con 0x7fb57415aa00 2024-04-25T11:19:55.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5741524b0 con 0x7fb57415aa00 2024-04-25T11:19:55.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5740ccc70 con 0x7fb57415aa00 2024-04-25T11:19:55.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb56c0b8670 con 0x7fb57415aa00 2024-04-25T11:19:55.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fb56c0ba7b0 con 0x7fb57415aa00 2024-04-25T11:19:55.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.000+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb57406a350 con 0x7fb57415aa00 2024-04-25T11:19:55.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.004+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb56c0dbe50 con 0x7fb57415aa00 2024-04-25T11:19:55.026 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.028+0000 7feb9a04c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:19:55.070 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.071+0000 7f9cdc3c1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T11:19:55.073 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.072+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4b178c77-9ab9-43ea-842b-66df2de40110", "id": 1} v 0) -- 0x7fb57415e210 con 0x7fb57415aa00 2024-04-25T11:19:55.080 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb56b7fe640 1 -- 172.21.15.157:0/1232991969 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b178c77-9ab9-43ea-842b-66df2de40110", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fb56c0ca070 con 0x7fb57415aa00 2024-04-25T11:19:55.080 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 msgr2=0x7fb574157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb574157b90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb5740c7c90 tx=0x7fb56c01bfa0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 shutdown_connections 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb57415aa00 0x7fb574157b90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb57415a5f0 0x7fb57414e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 --2- 172.21.15.157:0/1232991969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb574154060 0x7fb5741532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 >> 172.21.15.157:0/1232991969 conn(0x7fb5740ca5a0 msgr2=0x7fb5740c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 shutdown_connections 2024-04-25T11:19:55.082 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.080+0000 7fb5797ee640 1 -- 172.21.15.157:0/1232991969 wait complete. 2024-04-25T11:19:55.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 8eb9fd06-e438-4799-a0d9-3a3ed2faa796 2 2024-04-25T11:19:55.242 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 Processor -- start 2024-04-25T11:19:55.242 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 -- start start 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3c0015a5f0 0x7f3c000c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3c0015aa00 0x7f3c000c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c000c3880 con 0x7f3c00154060 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3c000c8fd0 con 0x7f3c0015a5f0 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c073aa640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3c0006d9b0 con 0x7f3c0015aa00 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3c0015aa00 0x7f3c000c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3c0015aa00 0x7f3c000c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55182/0 (socket says 172.21.15.157:55182) 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 -- 172.21.15.157:0/1458270420 learned_addr learned my addr 172.21.15.157:0/1458270420 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c04945640 1 --2- 172.21.15.157:0/1458270420 >> v2:172.21.15.136:3300/0 conn(0x7f3c0015a5f0 0x7f3c000c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.244 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 -- 172.21.15.157:0/1458270420 >> v2:172.21.15.136:3300/0 conn(0x7f3c0015a5f0 msgr2=0x7f3c000c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:55.245 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 --2- 172.21.15.157:0/1458270420 >> v2:172.21.15.136:3300/0 conn(0x7f3c0015a5f0 0x7f3c000c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.245 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 -- 172.21.15.157:0/1458270420 >> v1:172.21.15.136:6789/0 conn(0x7f3c00154060 legacy=0x7f3c000c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.245 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.244+0000 7f3c05947640 1 -- 172.21.15.157:0/1458270420 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c00154470 con 0x7f3c0015aa00 2024-04-25T11:19:55.246 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c05947640 1 --2- 172.21.15.157:0/1458270420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015aa00 0x7f3c000c76f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c00154640 tx=0x7f3bf801ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cf1706dea532923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.246 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3bf7fff640 1 -- 172.21.15.157:0/1458270420 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80b5070 con 0x7f3c0015aa00 2024-04-25T11:19:55.246 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3bf7fff640 1 -- 172.21.15.157:0/1458270420 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf80a8a10 con 0x7f3c0015aa00 2024-04-25T11:19:55.246 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3bf7fff640 1 -- 172.21.15.157:0/1458270420 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80a8de0 con 0x7f3c0015aa00 2024-04-25T11:19:55.247 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- 172.21.15.157:0/1458270420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015aa00 msgr2=0x7f3c000c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.247 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 --2- 172.21.15.157:0/1458270420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015aa00 0x7f3c000c76f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3c00154640 tx=0x7f3bf801ed10 comp rx=0 tx=0).stop 2024-04-25T11:19:55.247 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- 172.21.15.157:0/1458270420 shutdown_connections 2024-04-25T11:19:55.247 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 --2- 172.21.15.157:0/1458270420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015aa00 0x7f3c000c76f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.247 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 --2- 172.21.15.157:0/1458270420 >> v2:172.21.15.136:3300/0 conn(0x7f3c0015a5f0 0x7f3c000c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- 172.21.15.157:0/1458270420 >> 172.21.15.157:0/1458270420 conn(0x7f3c000ca5a0 msgr2=0x7f3c000ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- 172.21.15.157:0/1458270420 shutdown_connections 2024-04-25T11:19:55.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- 172.21.15.157:0/1458270420 wait complete. 2024-04-25T11:19:55.249 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 Processor -- start 2024-04-25T11:19:55.249 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.248+0000 7f3c073aa640 1 -- start start 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 0x7f3c00144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 0x7f3c0015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0015a0d0 con 0x7f3c00154060 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0006df30 con 0x7f3c0015a5f0 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3c00159d80 con 0x7f3c0015aa00 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c04945640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 0x7f3c0015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c04945640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55192/0 (socket says 172.21.15.157:55192) 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c04945640 1 -- 172.21.15.157:0/4285374934 learned_addr learned my addr 172.21.15.157:0/4285374934 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05146640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 0x7f3c00144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 -- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 msgr2=0x7f3c0015d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 -- 172.21.15.157:0/4285374934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 msgr2=0x7f3c00144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 0x7f3c00144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 -- 172.21.15.157:0/4285374934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c000737d0 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c04945640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05146640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 0x7f3c00144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c05947640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 0x7f3c0015dea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80af000 tx=0x7f3bf801bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80b5070 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf80b0a90 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80bb410 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c00146680 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c000cc2b0 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3bf80c2060 con 0x7f3c0015aa00 2024-04-25T11:19:55.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3bf80ba7e0 con 0x7f3c0015aa00 2024-04-25T11:19:55.254 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.252+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0006a350 con 0x7f3c0015aa00 2024-04-25T11:19:55.254 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.256+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3bf8018870 con 0x7f3c0015aa00 2024-04-25T11:19:55.265 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.267+0000 7f9cdc3c1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T11:19:55.323 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.324+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8eb9fd06-e438-4799-a0d9-3a3ed2faa796", "id": 2} v 0) -- 0x7f3c000c85e0 con 0x7f3c0015aa00 2024-04-25T11:19:55.330 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3bf5ffb640 1 -- 172.21.15.157:0/4285374934 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8eb9fd06-e438-4799-a0d9-3a3ed2faa796", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3bf80ba3c0 con 0x7f3c0015aa00 2024-04-25T11:19:55.330 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-04-25T11:19:55.331 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 msgr2=0x7f3c0015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 0x7f3c0015dea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80af000 tx=0x7f3bf801bab0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 shutdown_connections 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3c0015aa00 0x7f3c0015dea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3c0015a5f0 0x7f3c0015d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 --2- 172.21.15.157:0/4285374934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c00154060 0x7f3c00144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 >> 172.21.15.157:0/4285374934 conn(0x7f3c000ca5a0 msgr2=0x7f3c001523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 shutdown_connections 2024-04-25T11:19:55.332 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.332+0000 7f3c073aa640 1 -- 172.21.15.157:0/4285374934 wait complete. 2024-04-25T11:19:55.349 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 63bae1d0-386d-4a9f-9f2e-9db121da837d 3 2024-04-25T11:19:55.495 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 Processor -- start 2024-04-25T11:19:55.495 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 -- start start 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1c158ce0 con 0x7f0d1c0cb010 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0d1c073fc0 con 0x7f0d1c074b60 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d22ff2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c158f80 con 0x7f0d1c073230 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d2158f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d2158f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55196/0 (socket says 172.21.15.157:55196) 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47470/0 (socket says 172.21.15.157:47470) 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d2158f640 1 -- 172.21.15.157:0/2315714446 learned_addr learned my addr 172.21.15.157:0/2315714446 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d1bfff640 1 --1- 172.21.15.157:0/2315714446 >> v1:172.21.15.157:6789/0 conn(0x7f0d1c0cb010 0x7f0d1c074450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:57372/0 (socket says 172.21.15.157:57372) 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 -- 172.21.15.157:0/2315714446 >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 msgr2=0x7f0d1c163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 --2- 172.21.15.157:0/2315714446 >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 -- 172.21.15.157:0/2315714446 >> v1:172.21.15.157:6789/0 conn(0x7f0d1c0cb010 legacy=0x7f0d1c074450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d20d8e640 1 -- 172.21.15.157:0/2315714446 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1c159260 con 0x7f0d1c073230 2024-04-25T11:19:55.498 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.496+0000 7f0d2158f640 1 --2- 172.21.15.157:0/2315714446 >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.499 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d20d8e640 1 --2- 172.21.15.157:0/2315714446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009d90 tx=0x7f0d1001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dfbc1c2deef288d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.499 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d1b7fe640 1 -- 172.21.15.157:0/2315714446 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d100b2070 con 0x7f0d1c073230 2024-04-25T11:19:55.499 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d1b7fe640 1 -- 172.21.15.157:0/2315714446 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d100a5920 con 0x7f0d1c073230 2024-04-25T11:19:55.499 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d1b7fe640 1 -- 172.21.15.157:0/2315714446 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d100a5cf0 con 0x7f0d1c073230 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 -- 172.21.15.157:0/2315714446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 msgr2=0x7f0d1c0caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/2315714446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009d90 tx=0x7f0d1001c970 comp rx=0 tx=0).stop 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 -- 172.21.15.157:0/2315714446 shutdown_connections 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/2315714446 >> v2:172.21.15.157:3300/0 conn(0x7f0d1c074b60 0x7f0d1c163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/2315714446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c073230 0x7f0d1c0caad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 -- 172.21.15.157:0/2315714446 >> 172.21.15.157:0/2315714446 conn(0x7f0d1c06f730 msgr2=0x7f0d1c06fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 -- 172.21.15.157:0/2315714446 shutdown_connections 2024-04-25T11:19:55.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.500+0000 7f0d22ff2640 1 -- 172.21.15.157:0/2315714446 wait complete. 2024-04-25T11:19:55.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 Processor -- start 2024-04-25T11:19:55.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- start start 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 0x7f0d1c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 0x7f0d1c15c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1c0c2ba0 con 0x7f0d1c073230 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1c1532f0 con 0x7f0d1c074b60 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c06dd80 con 0x7f0d1c0cb010 2024-04-25T11:19:55.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d1bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 0x7f0d1c15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47478/0 (socket says 172.21.15.157:47478) 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 -- 172.21.15.157:0/1176586842 learned_addr learned my addr 172.21.15.157:0/1176586842 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d20d8e640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 0x7f0d1c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 -- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 msgr2=0x7f0d1c15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 0x7f0d1c15c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 -- 172.21.15.157:0/1176586842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 msgr2=0x7f0d1c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 0x7f0d1c0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 -- 172.21.15.157:0/1176586842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1c0cd380 con 0x7f0d1c0cb010 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d1bfff640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 0x7f0d1c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d20d8e640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 0x7f0d1c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:19:55.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d2158f640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400e890 tx=0x7f0d1400ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14004030 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d140041d0 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1c1f3e10 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1400c940 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d1c1f43d0 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d14003620 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f0d14003da0 con 0x7f0d1c0cb010 2024-04-25T11:19:55.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.504+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d1c06a350 con 0x7f0d1c0cb010 2024-04-25T11:19:55.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.508+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0d14017020 con 0x7f0d1c0cb010 2024-04-25T11:19:55.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.576+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "63bae1d0-386d-4a9f-9f2e-9db121da837d", "id": 3} v 0) -- 0x7f0d1c1637c0 con 0x7f0d1c0cb010 2024-04-25T11:19:55.584 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.584+0000 7f0d197fa640 1 -- 172.21.15.157:0/1176586842 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "63bae1d0-386d-4a9f-9f2e-9db121da837d", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f0d14003910 con 0x7f0d1c0cb010 2024-04-25T11:19:55.584 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 msgr2=0x7f0d1c06c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400e890 tx=0x7f0d1400ed50 comp rx=0 tx=0).stop 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 shutdown_connections 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d1c0cb010 0x7f0d1c06c670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d1c074b60 0x7f0d1c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 --2- 172.21.15.157:0/1176586842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d1c073230 0x7f0d1c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 >> 172.21.15.157:0/1176586842 conn(0x7f0d1c06f730 msgr2=0x7f0d1c0c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.587 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 shutdown_connections 2024-04-25T11:19:55.587 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.588+0000 7f0d22ff2640 1 -- 172.21.15.157:0/1176586842 wait complete. 2024-04-25T11:19:55.598 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.600+0000 7feb9a04c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:19:55.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new c71fefe5-e003-4765-b8ac-accf933992ad 4 2024-04-25T11:19:55.655 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.656+0000 7feb9a04c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:19:55.711 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.712+0000 7feb9a04c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:19:55.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.752+0000 7fdb6c94a640 1 Processor -- start 2024-04-25T11:19:55.752 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.752+0000 7fdb6c94a640 1 -- start start 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdb680ca290 0x7fdb680ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdb6815d750 con 0x7fdb680ca290 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdb6815da30 con 0x7fdb680b5060 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6c94a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6815dd10 con 0x7fdb6815a430 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.755 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55202/0 (socket says 172.21.15.157:55202) 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47480/0 (socket says 172.21.15.157:47480) 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6659c640 1 -- 172.21.15.157:0/3880438970 learned_addr learned my addr 172.21.15.157:0/3880438970 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb65d9b640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.136:3300/0 conn(0x7fdb680ca290 0x7fdb680ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 -- 172.21.15.157:0/3880438970 >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 msgr2=0x7fdb680c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 -- 172.21.15.157:0/3880438970 >> v2:172.21.15.136:3300/0 conn(0x7fdb680ca290 msgr2=0x7fdb680ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.136:3300/0 conn(0x7fdb680ca290 0x7fdb680ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 -- 172.21.15.157:0/3880438970 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6815dff0 con 0x7fdb6815a430 2024-04-25T11:19:55.756 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6659c640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.757 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/3880438970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb68068ff0 tx=0x7fdb5801f110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9936789f29b4d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.757 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6559a640 1 -- 172.21.15.157:0/3880438970 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb580b5070 con 0x7fdb6815a430 2024-04-25T11:19:55.757 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6559a640 1 -- 172.21.15.157:0/3880438970 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb580b0440 con 0x7fdb6815a430 2024-04-25T11:19:55.757 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.756+0000 7fdb6559a640 1 -- 172.21.15.157:0/3880438970 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb580b0810 con 0x7fdb6815a430 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- 172.21.15.157:0/3880438970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 msgr2=0x7fdb6815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/3880438970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb68068ff0 tx=0x7fdb5801f110 comp rx=0 tx=0).stop 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- 172.21.15.157:0/3880438970 shutdown_connections 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.136:3300/0 conn(0x7fdb680ca290 0x7fdb680ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/3880438970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb6815a430 0x7fdb6815a840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/3880438970 >> v2:172.21.15.157:3300/0 conn(0x7fdb680b5060 0x7fdb680c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.758 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- 172.21.15.157:0/3880438970 >> 172.21.15.157:0/3880438970 conn(0x7fdb68070f70 msgr2=0x7fdb68068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.759 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- 172.21.15.157:0/3880438970 shutdown_connections 2024-04-25T11:19:55.759 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- 172.21.15.157:0/3880438970 wait complete. 2024-04-25T11:19:55.759 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 Processor -- start 2024-04-25T11:19:55.759 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- start start 2024-04-25T11:19:55.762 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.762 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 0x7fdb681f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.762 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 0x7fdb680b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.762 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdb6815ad80 con 0x7fdb680ca290 2024-04-25T11:19:55.762 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdb68159f10 con 0x7fdb6815a430 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6c94a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb6815d3f0 con 0x7fdb680b5060 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47496/0 (socket says 172.21.15.157:47496) 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 0x7fdb680b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6659c640 1 -- 172.21.15.157:0/4022997805 learned_addr learned my addr 172.21.15.157:0/4022997805 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb65d9b640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 0x7fdb681f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 -- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 msgr2=0x7fdb680b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 -- 172.21.15.157:0/4022997805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 msgr2=0x7fdb681f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 0x7fdb681f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb66d9d640 1 -- 172.21.15.157:0/4022997805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb680cd300 con 0x7fdb6815a430 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb6659c640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.760+0000 7fdb65d9b640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 0x7fdb681f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb66d9d640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 0x7fdb680b3290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb580af000 tx=0x7fdb5801beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:55.763 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb580b5050 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb580c1070 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58009450 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb680cd5e0 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb680cdba0 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdb580bc890 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fdb580c6d30 con 0x7fdb6815a430 2024-04-25T11:19:55.764 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb6806a350 con 0x7fdb6815a430 2024-04-25T11:19:55.765 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.764+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdb58018c70 con 0x7fdb6815a430 2024-04-25T11:19:55.792 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.792+0000 7feb9a04c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:19:55.810 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.811+0000 7f9cdc3c1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T11:19:55.833 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.832+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c71fefe5-e003-4765-b8ac-accf933992ad", "id": 4} v 0) -- 0x7fdb680cadd0 con 0x7fdb6815a430 2024-04-25T11:19:55.839 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb56ffd640 1 -- 172.21.15.157:0/4022997805 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c71fefe5-e003-4765-b8ac-accf933992ad", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fdb580bccc0 con 0x7fdb6815a430 2024-04-25T11:19:55.840 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-04-25T11:19:55.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 msgr2=0x7fdb680b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:55.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 0x7fdb680b3290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdb580af000 tx=0x7fdb5801beb0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 shutdown_connections 2024-04-25T11:19:55.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb6815a430 0x7fdb680b3290 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.841 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb680ca290 0x7fdb681f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 --2- 172.21.15.157:0/4022997805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb680b5060 0x7fdb680b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:55.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 >> 172.21.15.157:0/4022997805 conn(0x7fdb68070f70 msgr2=0x7fdb6814a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:55.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 shutdown_connections 2024-04-25T11:19:55.842 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.840+0000 7fdb6c94a640 1 -- 172.21.15.157:0/4022997805 wait complete. 2024-04-25T11:19:55.846 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.848+0000 7feb9a04c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:19:55.859 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 3a3aa176-27ff-4960-b063-b0b3c6c4daa6 5 2024-04-25T11:19:55.864 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.863+0000 7f9cdc3c1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T11:19:55.916 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.915+0000 7f9cdc3c1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T11:19:55.928 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.928+0000 7feb9a04c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:19:55.983 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:55.984+0000 7feb9a04c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:19:55.992 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:55.995+0000 7f9cdc3c1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T11:19:55.997 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.996+0000 7f3a3fac4640 1 Processor -- start 2024-04-25T11:19:55.997 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:55.996+0000 7f3a3fac4640 1 -- start start 2024-04-25T11:19:55.999 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3fac4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a380c90c0 0x7f3a38167a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:55.999 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3fac4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a381605b0 con 0x7f3a3815ccb0 2024-04-25T11:19:55.999 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3fac4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a381602d0 con 0x7f3a380729c0 2024-04-25T11:19:55.999 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3fac4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3815c820 con 0x7f3a380c90c0 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3e061640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a380c90c0 0x7f3a38167a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3d860640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3a380729c0 0x7f3a38156210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:57382/0 (socket says 172.21.15.157:57382) 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3d860640 1 -- 172.21.15.157:0/1202109417 learned_addr learned my addr 172.21.15.157:0/1202109417 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3e061640 1 -- 172.21.15.157:0/1202109417 >> v1:172.21.15.157:6789/0 conn(0x7f3a380729c0 legacy=0x7f3a38156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3e061640 1 -- 172.21.15.157:0/1202109417 >> v1:172.21.15.136:6789/0 conn(0x7f3a3815ccb0 legacy=0x7f3a380c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3c85e640 1 -- 172.21.15.157:0/1202109417 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455688190 0 0) 0x7f3a381602d0 con 0x7f3a380729c0 2024-04-25T11:19:56.000 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.000+0000 7f3a3e061640 1 -- 172.21.15.157:0/1202109417 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3815c1d0 con 0x7f3a380c90c0 2024-04-25T11:19:56.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3e061640 1 --2- 172.21.15.157:0/1202109417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a380c90c0 0x7f3a38167a50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009d90 tx=0x7f3a3001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74734626263978ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3c85e640 1 -- 172.21.15.157:0/1202109417 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300b2070 con 0x7f3a380c90c0 2024-04-25T11:19:56.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3c85e640 1 -- 172.21.15.157:0/1202109417 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a300a5830 con 0x7f3a380c90c0 2024-04-25T11:19:56.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3c85e640 1 -- 172.21.15.157:0/1202109417 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300a5c00 con 0x7f3a380c90c0 2024-04-25T11:19:56.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- 172.21.15.157:0/1202109417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a380c90c0 msgr2=0x7f3a38167a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/1202109417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a380c90c0 0x7f3a38167a50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009d90 tx=0x7f3a3001c970 comp rx=0 tx=0).stop 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- 172.21.15.157:0/1202109417 shutdown_connections 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/1202109417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a380c90c0 0x7f3a38167a50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- 172.21.15.157:0/1202109417 >> 172.21.15.157:0/1202109417 conn(0x7f3a380c9d50 msgr2=0x7f3a380ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- 172.21.15.157:0/1202109417 shutdown_connections 2024-04-25T11:19:56.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- 172.21.15.157:0/1202109417 wait complete. 2024-04-25T11:19:56.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 Processor -- start 2024-04-25T11:19:56.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.004+0000 7f3a3fac4640 1 -- start start 2024-04-25T11:19:56.008 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.008 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a380c90c0 0x7f3a3815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.008 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3816a7b0 con 0x7f3a380c90c0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3815d0c0 con 0x7f3a3815ccb0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3816a930 con 0x7f3a380729c0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3e061640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47502/0 (socket says 172.21.15.157:47502) 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3e061640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55232/0 (socket says 172.21.15.157:55232) 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 -- 172.21.15.157:0/3940112609 learned_addr learned my addr 172.21.15.157:0/3940112609 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d05f640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a380c90c0 0x7f3a3815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 -- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 msgr2=0x7f3a38146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.009 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 -- 172.21.15.157:0/3940112609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a380c90c0 msgr2=0x7f3a3815ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a380c90c0 0x7f3a3815ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 -- 172.21.15.157:0/3940112609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a380cca20 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3e061640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3d860640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c440 tx=0x7f3a3400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340074e0 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a340047f0 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3815e560 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34007680 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a380cccc0 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a34011070 con 0x7f3a380729c0 2024-04-25T11:19:56.010 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f3a34013940 con 0x7f3a380729c0 2024-04-25T11:19:56.011 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.008+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3806a350 con 0x7f3a380729c0 2024-04-25T11:19:56.012 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.012+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3a34013c60 con 0x7f3a380729c0 2024-04-25T11:19:56.044 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:56.047+0000 7f9cdc3c1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T11:19:56.081 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.080+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3a3aa176-27ff-4960-b063-b0b3c6c4daa6", "id": 5} v 0) -- 0x7f3a3815d7b0 con 0x7f3a380729c0 2024-04-25T11:19:56.089 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.088+0000 7f3a2e7fc640 1 -- 172.21.15.157:0/3940112609 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3a3aa176-27ff-4960-b063-b0b3c6c4daa6", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f3a340133c0 con 0x7f3a380729c0 2024-04-25T11:19:56.089 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-04-25T11:19:56.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 msgr2=0x7f3a380cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c440 tx=0x7f3a3400c900 comp rx=0 tx=0).stop 2024-04-25T11:19:56.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 shutdown_connections 2024-04-25T11:19:56.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a3815ccb0 0x7f3a38146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.090 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a380c90c0 0x7f3a3815ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 --2- 172.21.15.157:0/3940112609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a380729c0 0x7f3a380cc2b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 >> 172.21.15.157:0/3940112609 conn(0x7f3a380c9d50 msgr2=0x7f3a38156210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 shutdown_connections 2024-04-25T11:19:56.091 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.092+0000 7f3a3fac4640 1 -- 172.21.15.157:0/3940112609 wait complete. 2024-04-25T11:19:56.108 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 4eb1c4e4-121a-4669-b755-7f86a53e2e23 6 2024-04-25T11:19:56.122 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:56.123+0000 7f9cdc3c1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T11:19:56.143 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:56.144+0000 7feb9a04c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:19:56.175 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:56.175+0000 7f9cdc3c1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T11:19:56.207 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:56.208+0000 7feb9a04c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:19:56.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 Processor -- start 2024-04-25T11:19:56.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 -- start start 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faf0815ff60 0x7faf080ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0815c6f0 con 0x7faf08160370 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7faf080c6310 con 0x7faf0815ff60 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ece5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf0815c990 con 0x7faf0815c0d0 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ca81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ca81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:47518/0 (socket says 172.21.15.157:47518) 2024-04-25T11:19:56.250 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.248+0000 7faf0ca81640 1 -- 172.21.15.157:0/3534170220 learned_addr learned my addr 172.21.15.157:0/3534170220 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:56.251 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf07fff640 1 --2- 172.21.15.157:0/3534170220 >> v2:172.21.15.136:3300/0 conn(0x7faf0815ff60 0x7faf080ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.251 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ca81640 1 -- 172.21.15.157:0/3534170220 >> v2:172.21.15.136:3300/0 conn(0x7faf0815ff60 msgr2=0x7faf080ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:56.251 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ca81640 1 --2- 172.21.15.157:0/3534170220 >> v2:172.21.15.136:3300/0 conn(0x7faf0815ff60 0x7faf080ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.251 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ca81640 1 -- 172.21.15.157:0/3534170220 >> v1:172.21.15.136:6789/0 conn(0x7faf08160370 legacy=0x7faf08071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:56.251 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ca81640 1 -- 172.21.15.157:0/3534170220 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0815cc30 con 0x7faf0815c0d0 2024-04-25T11:19:56.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ca81640 1 --2- 172.21.15.157:0/3534170220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faefc009870 tx=0x7faefc01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a865a07f2e841ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf077fe640 1 -- 172.21.15.157:0/3534170220 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0b5070 con 0x7faf0815c0d0 2024-04-25T11:19:56.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf077fe640 1 -- 172.21.15.157:0/3534170220 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faefc0a89d0 con 0x7faf0815c0d0 2024-04-25T11:19:56.252 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf077fe640 1 -- 172.21.15.157:0/3534170220 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0a8da0 con 0x7faf0815c0d0 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 -- 172.21.15.157:0/3534170220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 msgr2=0x7faf080cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 --2- 172.21.15.157:0/3534170220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faefc009870 tx=0x7faefc01ed10 comp rx=0 tx=0).stop 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 -- 172.21.15.157:0/3534170220 shutdown_connections 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 --2- 172.21.15.157:0/3534170220 >> v2:172.21.15.136:3300/0 conn(0x7faf0815ff60 0x7faf080ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 --2- 172.21.15.157:0/3534170220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815c0d0 0x7faf080cb720 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 -- 172.21.15.157:0/3534170220 >> 172.21.15.157:0/3534170220 conn(0x7faf0806f9d0 msgr2=0x7faf0806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 -- 172.21.15.157:0/3534170220 shutdown_connections 2024-04-25T11:19:56.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.252+0000 7faf0ece5640 1 -- 172.21.15.157:0/3534170220 wait complete. 2024-04-25T11:19:56.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 Processor -- start 2024-04-25T11:19:56.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- start start 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 0x7faf08146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 0x7faf08152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faf080cbc60 con 0x7faf08160370 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf0815fa40 con 0x7faf0815c0d0 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf0815cf10 con 0x7faf0815ff60 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ca81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 0x7faf08146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ca81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55238/0 (socket says 172.21.15.157:55238) 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ca81640 1 -- 172.21.15.157:0/1616561647 learned_addr learned my addr 172.21.15.157:0/1616561647 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:56.258 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0d282640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 0x7faf08152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 -- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 msgr2=0x7faf081f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 -- 172.21.15.157:0/1616561647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 msgr2=0x7faf08152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 0x7faf08152870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 -- 172.21.15.157:0/1616561647 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf08155870 con 0x7faf0815ff60 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ca81640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0d282640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 0x7faf08152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf07fff640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 0x7faf08146c60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faf080caf60 tx=0x7faef400da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef40125f0 con 0x7faf0815ff60 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef4019070 con 0x7faf0815ff60 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf08155b50 con 0x7faf0815ff60 2024-04-25T11:19:56.259 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef40129d0 con 0x7faf0815ff60 2024-04-25T11:19:56.260 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf081f3470 con 0x7faf0815ff60 2024-04-25T11:19:56.260 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faef4012b70 con 0x7faf0815ff60 2024-04-25T11:19:56.260 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.256+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7faef401ea50 con 0x7faf0815ff60 2024-04-25T11:19:56.260 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.260+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf081555c0 con 0x7faf0815ff60 2024-04-25T11:19:56.261 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.260+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7faef401e460 con 0x7faf0815ff60 2024-04-25T11:19:56.323 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:56.323+0000 7f9cdc3c1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T11:19:56.325 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.324+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4eb1c4e4-121a-4669-b755-7f86a53e2e23", "id": 6} v 0) -- 0x7faf0806a350 con 0x7faf0815ff60 2024-04-25T11:19:56.334 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf057fa640 1 -- 172.21.15.157:0/1616561647 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4eb1c4e4-121a-4669-b755-7f86a53e2e23", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7faef401e640 con 0x7faf0815ff60 2024-04-25T11:19:56.334 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-04-25T11:19:56.335 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 msgr2=0x7faf08146c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 0x7faf08146c60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faf080caf60 tx=0x7faef400da30 comp rx=0 tx=0).stop 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 shutdown_connections 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf08160370 0x7faf08152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf0815ff60 0x7faf08146c60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 --2- 172.21.15.157:0/1616561647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0815c0d0 0x7faf081f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 >> 172.21.15.157:0/1616561647 conn(0x7faf0806f9d0 msgr2=0x7faf08148cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 shutdown_connections 2024-04-25T11:19:56.336 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.336+0000 7faf0ece5640 1 -- 172.21.15.157:0/1616561647 wait complete. 2024-04-25T11:19:56.352 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 6736762c-62fd-4145-8a2d-91f485a23404 7 2024-04-25T11:19:56.386 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:56.387+0000 7f9cdc3c1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T11:19:56.550 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.548+0000 7f8d5e455640 1 Processor -- start 2024-04-25T11:19:56.550 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.548+0000 7f8d5e455640 1 -- start start 2024-04-25T11:19:56.550 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.548+0000 7f8d5e455640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d5806f910 0x7f8d580c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.548+0000 7f8d5e455640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8d580c29f0 0x7f8d5816ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5e455640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d58160270 con 0x7f8d580c3a90 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5e455640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8d580c9820 con 0x7f8d580c29f0 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5e455640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d580c92a0 con 0x7f8d5806f910 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8d580c29f0 0x7f8d5816ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8d580c29f0 0x7f8d5816ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55254/0 (socket says 172.21.15.157:55254) 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 -- 172.21.15.157:0/3280888267 learned_addr learned my addr 172.21.15.157:0/3280888267 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:56.551 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d57fff640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d5806f910 0x7f8d580c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.552 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 -- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d5806f910 msgr2=0x7f8d580c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:56.552 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d5806f910 0x7f8d580c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.552 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 -- 172.21.15.157:0/3280888267 >> v1:172.21.15.136:6789/0 conn(0x7f8d580c3a90 legacy=0x7f8d58073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:56.552 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.552+0000 7f8d5c9f2640 1 -- 172.21.15.157:0/3280888267 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5806ef20 con 0x7f8d580c29f0 2024-04-25T11:19:56.555 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.556+0000 7f8d5c9f2640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d580c29f0 0x7f8d5816ab10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009e50 tx=0x7f8d5001c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca365be244be953c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d56ffd640 1 -- 172.21.15.157:0/3280888267 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d500a5710 con 0x7f8d580c29f0 2024-04-25T11:19:56.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d56ffd640 1 -- 172.21.15.157:0/3280888267 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d500a58b0 con 0x7f8d580c29f0 2024-04-25T11:19:56.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d56ffd640 1 -- 172.21.15.157:0/3280888267 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d500a5c80 con 0x7f8d580c29f0 2024-04-25T11:19:56.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 -- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d580c29f0 msgr2=0x7f8d5816ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d580c29f0 0x7f8d5816ab10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009e50 tx=0x7f8d5001c980 comp rx=0 tx=0).stop 2024-04-25T11:19:56.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 -- 172.21.15.157:0/3280888267 shutdown_connections 2024-04-25T11:19:56.568 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d580c29f0 0x7f8d5816ab10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.568 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 --2- 172.21.15.157:0/3280888267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d5806f910 0x7f8d580c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.568 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 -- 172.21.15.157:0/3280888267 >> 172.21.15.157:0/3280888267 conn(0x7f8d580cdae0 msgr2=0x7f8d580c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.568 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 -- 172.21.15.157:0/3280888267 shutdown_connections 2024-04-25T11:19:56.568 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.568+0000 7f8d5e455640 1 -- 172.21.15.157:0/3280888267 wait complete. 2024-04-25T11:19:56.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.572+0000 7f8d5e455640 1 Processor -- start 2024-04-25T11:19:56.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.572+0000 7f8d5e455640 1 -- start start 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 0x7f8d58159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d580c3a90 0x7f8d581fce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d580743b0 con 0x7f8d580c29f0 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d58074230 con 0x7f8d5806f910 2024-04-25T11:19:56.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d58074fd0 con 0x7f8d580c3a90 2024-04-25T11:19:56.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:55260/0 (socket says 172.21.15.157:55260) 2024-04-25T11:19:56.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 -- 172.21.15.157:0/3733098327 learned_addr learned my addr 172.21.15.157:0/3733098327 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-04-25T11:19:56.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5c9f2640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d580c3a90 0x7f8d581fce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d577fe640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 0x7f8d58159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 -- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d580c3a90 msgr2=0x7f8d581fce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d580c3a90 0x7f8d581fce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 -- 172.21.15.157:0/3733098327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 msgr2=0x7f8d58159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 0x7f8d58159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 -- 172.21.15.157:0/3733098327 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d581fa520 con 0x7f8d5806f910 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d577fe640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 0x7f8d58159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d57fff640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800e8a0 tx=0x7f8d4800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48017070 con 0x7f8d5806f910 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4800c940 con 0x7f8d5806f910 2024-04-25T11:19:56.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 -- 172.21.15.157:0/3733098327 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d58153940 con 0x7f8d5806f910 2024-04-25T11:19:56.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4800ccf0 con 0x7f8d5806f910 2024-04-25T11:19:56.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d5e455640 1 -- 172.21.15.157:0/3733098327 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d581f97e0 con 0x7f8d5806f910 2024-04-25T11:19:56.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d4801a070 con 0x7f8d5806f910 2024-04-25T11:19:56.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.576+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f8d48013ad0 con 0x7f8d5806f910 2024-04-25T11:19:56.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.580+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d581fdb20 con 0x7f8d5806f910 2024-04-25T11:19:56.583 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.584+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8d48013470 con 0x7f8d5806f910 2024-04-25T11:19:56.652 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.652+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6736762c-62fd-4145-8a2d-91f485a23404", "id": 7} v 0) -- 0x7f8d5806a2e0 con 0x7f8d5806f910 2024-04-25T11:19:56.659 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d54ff9640 1 -- 172.21.15.157:0/3733098327 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6736762c-62fd-4145-8a2d-91f485a23404", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f8d48013650 con 0x7f8d5806f910 2024-04-25T11:19:56.659 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-04-25T11:19:56.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 msgr2=0x7f8d581f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800e8a0 tx=0x7f8d4800eca0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 shutdown_connections 2024-04-25T11:19:56.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d580c3a90 0x7f8d581fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.660 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d580c29f0 0x7f8d58159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.661 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 --2- 172.21.15.157:0/3733098327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d5806f910 0x7f8d581f9fe0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.661 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 >> 172.21.15.157:0/3733098327 conn(0x7f8d580cdae0 msgr2=0x7f8d5815f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.661 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 shutdown_connections 2024-04-25T11:19:56.661 INFO:teuthology.orchestra.run.smithi157.stderr:2024-04-25T11:19:56.660+0000 7f8d3e7fc640 1 -- 172.21.15.157:0/3733098327 wait complete. 2024-04-25T11:19:56.677 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T11:19:56.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T11:19:56.680 INFO:tasks.ceph.osd.0:Started 2024-04-25T11:19:56.680 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T11:19:56.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T11:19:56.682 INFO:tasks.ceph.osd.1:Started 2024-04-25T11:19:56.683 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T11:19:56.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T11:19:56.685 INFO:tasks.ceph.osd.2:Started 2024-04-25T11:19:56.685 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T11:19:56.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T11:19:56.687 INFO:tasks.ceph.osd.3:Started 2024-04-25T11:19:56.687 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T11:19:56.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T11:19:56.689 INFO:tasks.ceph.osd.4:Started 2024-04-25T11:19:56.690 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T11:19:56.690 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T11:19:56.691 INFO:tasks.ceph.osd.5:Started 2024-04-25T11:19:56.691 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T11:19:56.692 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T11:19:56.693 INFO:tasks.ceph.osd.6:Started 2024-04-25T11:19:56.693 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T11:19:56.694 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T11:19:56.695 INFO:tasks.ceph.osd.7:Started 2024-04-25T11:19:56.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:19:56.828 INFO:tasks.ceph.mgr.x.smithi157.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T11:19:56.828 INFO:tasks.ceph.mgr.x.smithi157.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T11:19:56.829 INFO:tasks.ceph.mgr.x.smithi157.stderr: from numpy import show_config as show_numpy_config 2024-04-25T11:19:56.867 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:56.868+0000 7feb9a04c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:19:56.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.907+0000 7f99bfacf640 1 Processor -- start 2024-04-25T11:19:56.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.907+0000 7f99bfacf640 1 -- start start 2024-04-25T11:19:56.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bfacf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bfacf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f99b815aa00 0x7f99b8164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bfacf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b80c8d40 con 0x7f99b815a5f0 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bfacf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f99b80c8fd0 con 0x7f99b815aa00 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bfacf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99b80c3880 con 0x7f99b8154060 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f99b815aa00 0x7f99b8164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bd86b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f99b815aa00 0x7f99b8164700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52724/0 (socket says 172.21.15.136:52724) 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 -- 172.21.15.136:0/3906940809 learned_addr learned my addr 172.21.15.136:0/3906940809 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bd86b640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:60880/0 (socket says 172.21.15.136:60880) 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bd06a640 1 --1- 172.21.15.136:0/3906940809 >> v1:172.21.15.157:6789/0 conn(0x7f99b815a5f0 0x7f99b80c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:57536/0 (socket says 172.21.15.136:57536) 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 -- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 msgr2=0x7f99b80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 -- 172.21.15.136:0/3906940809 >> v1:172.21.15.157:6789/0 conn(0x7f99b815a5f0 legacy=0x7f99b80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 -- 172.21.15.136:0/3906940809 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b806d9b0 con 0x7f99b815aa00 2024-04-25T11:19:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99bd86b640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:19:56.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.911+0000 7f99be06c640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815aa00 0x7f99b8164700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99b806db80 tx=0x7f99b001f200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442d5a2dea42a62e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bc869640 1 -- 172.21.15.136:0/3906940809 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00b4070 con 0x7f99b815aa00 2024-04-25T11:19:56.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bc869640 1 -- 172.21.15.136:0/3906940809 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b00af400 con 0x7f99b815aa00 2024-04-25T11:19:56.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bc869640 1 -- 172.21.15.136:0/3906940809 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00af7d0 con 0x7f99b815aa00 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815aa00 msgr2=0x7f99b8164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815aa00 0x7f99b8164700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99b806db80 tx=0x7f99b001f200 comp rx=0 tx=0).stop 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- 172.21.15.136:0/3906940809 shutdown_connections 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815aa00 0x7f99b8164700 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3906940809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b8154060 0x7f99b80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- 172.21.15.136:0/3906940809 >> 172.21.15.136:0/3906940809 conn(0x7f99b80ca5a0 msgr2=0x7f99b80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- 172.21.15.136:0/3906940809 shutdown_connections 2024-04-25T11:19:56.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- 172.21.15.136:0/3906940809 wait complete. 2024-04-25T11:19:56.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 Processor -- start 2024-04-25T11:19:56.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- start start 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 0x7f99b81e3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b815aa00 0x7f99b8070b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f99b806df30 con 0x7f99b8154060 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8154470 con 0x7f99b815a5f0 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bfacf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99b815a0d0 con 0x7f99b815aa00 2024-04-25T11:19:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bd86b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bd86b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49548/0 (socket says 172.21.15.136:49548) 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.915+0000 7f99bd86b640 1 -- 172.21.15.136:0/3729161610 learned_addr learned my addr 172.21.15.136:0/3729161610 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd06a640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 0x7f99b81e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99be06c640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b815aa00 0x7f99b8070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 -- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b815aa00 msgr2=0x7f99b8070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b815aa00 0x7f99b8070b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 -- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 msgr2=0x7f99b81e3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 0x7f99b81e3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 -- 172.21.15.136:0/3729161610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b8156f90 con 0x7f99b8154060 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd06a640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 0x7f99b81e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bd86b640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f99b400e980 tx=0x7f99b400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4017070 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b400ca00 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b400cdd0 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b8157250 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81554f0 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f99b401a070 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f99b401cb20 con 0x7f99b8154060 2024-04-25T11:19:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.919+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b81e5410 con 0x7f99b8154060 2024-04-25T11:19:56.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.923+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f99b81e5410 con 0x7f99b8154060 2024-04-25T11:19:56.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.974+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99b806a350 con 0x7f99b8154060 2024-04-25T11:19:56.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.974+0000 7f99ae7fc640 1 -- 172.21.15.136:0/3729161610 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f99b401c450 con 0x7f99b8154060 2024-04-25T11:19:56.975 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:19:56.975 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":10,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:19:56.659894+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 msgr2=0x7f99b8156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f99b400e980 tx=0x7f99b400ee40 comp rx=0 tx=0).stop 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 shutdown_connections 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99b815aa00 0x7f99b8070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99b815a5f0 0x7f99b81e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 --2- 172.21.15.136:0/3729161610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8154060 0x7f99b8156970 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:56.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 >> 172.21.15.136:0/3729161610 conn(0x7f99b80ca5a0 msgr2=0x7f99b814d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:56.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 shutdown_connections 2024-04-25T11:19:56.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:19:56.978+0000 7f99bfacf640 1 -- 172.21.15.136:0/3729161610 wait complete. 2024-04-25T11:19:56.993 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T11:19:56.993 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T11:19:57.012 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:57.012+0000 7feb9a04c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:19:57.046 INFO:tasks.ceph.mgr.y.smithi136.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T11:19:57.046 INFO:tasks.ceph.mgr.y.smithi136.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T11:19:57.047 INFO:tasks.ceph.mgr.y.smithi136.stderr: from numpy import show_config as show_numpy_config 2024-04-25T11:19:57.129 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:57.130+0000 7f9cdc3c1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T11:19:57.179 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:57.180+0000 7feb9a04c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:19:57.243 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:57.244+0000 7feb9a04c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:19:57.305 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:57.306+0000 7f9cdc3c1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:19:57.331 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-04-25T11:19:57.332+0000 7faf826d07c0 -1 Falling back to public interface 2024-04-25T11:19:57.347 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-04-25T11:19:57.348+0000 7feaf20a17c0 -1 Falling back to public interface 2024-04-25T11:19:57.347 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-04-25T11:19:57.348+0000 7f1554c257c0 -1 Falling back to public interface 2024-04-25T11:19:57.363 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-04-25T11:19:57.364+0000 7f6ed1b337c0 -1 Falling back to public interface 2024-04-25T11:19:57.365 INFO:tasks.ceph.osd.0.smithi136.stderr:2024-04-25T11:19:57.366+0000 7eff528037c0 -1 Falling back to public interface 2024-04-25T11:19:57.383 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:57.384+0000 7feb9a04c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:19:57.385 INFO:tasks.ceph.osd.3.smithi136.stderr:2024-04-25T11:19:57.386+0000 7fb5152bb7c0 -1 Falling back to public interface 2024-04-25T11:19:57.389 INFO:tasks.ceph.osd.2.smithi136.stderr:2024-04-25T11:19:57.390+0000 7fd0f7f207c0 -1 Falling back to public interface 2024-04-25T11:19:57.396 INFO:tasks.ceph.osd.1.smithi136.stderr:2024-04-25T11:19:57.398+0000 7fcd283e17c0 -1 Falling back to public interface 2024-04-25T11:19:57.469 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:57.470+0000 7f9cdc3c1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T11:19:57.532 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:57.534+0000 7f9cdc3c1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T11:19:57.665 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:57.666+0000 7f9cdc3c1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T11:19:57.799 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T11:19:57.941 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 Processor -- start 2024-04-25T11:19:57.941 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 -- start start 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7914070e00 0x7f7914071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f79140c7cd0 0x7f79140c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79140c9fd0 con 0x7f79140c8ca0 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f79140ca270 con 0x7f7914070e00 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791c492640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f79140ca550 con 0x7f79140c7cd0 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7914070e00 0x7f7914071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7914070e00 0x7f7914071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49564/0 (socket says 172.21.15.136:49564) 2024-04-25T11:19:57.943 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 -- 172.21.15.136:0/2693435324 learned_addr learned my addr 172.21.15.136:0/2693435324 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:57.944 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f791aa2f640 1 --2- 172.21.15.136:0/2693435324 >> v2:172.21.15.157:3300/0 conn(0x7f79140c7cd0 0x7f79140c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:57.944 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 -- 172.21.15.136:0/2693435324 >> v2:172.21.15.157:3300/0 conn(0x7f79140c7cd0 msgr2=0x7f79140c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:19:57.944 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 --2- 172.21.15.136:0/2693435324 >> v2:172.21.15.157:3300/0 conn(0x7f79140c7cd0 0x7f79140c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:57.944 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 -- 172.21.15.136:0/2693435324 >> v1:172.21.15.157:6789/0 conn(0x7f79140c8ca0 legacy=0x7f7914152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:57.944 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.942+0000 7f7919a2d640 1 -- 172.21.15.136:0/2693435324 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79140ca830 con 0x7f7914070e00 2024-04-25T11:19:57.946 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f7919a2d640 1 --2- 172.21.15.136:0/2693435324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7914070e00 0x7f7914071210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f79140c6d70 tx=0x7f790401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ef4e3e00380ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:57.946 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791922c640 1 -- 172.21.15.136:0/2693435324 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79040ab430 con 0x7f7914070e00 2024-04-25T11:19:57.946 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791922c640 1 -- 172.21.15.136:0/2693435324 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79040b0070 con 0x7f7914070e00 2024-04-25T11:19:57.946 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791922c640 1 -- 172.21.15.136:0/2693435324 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79040ab7f0 con 0x7f7914070e00 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 -- 172.21.15.136:0/2693435324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7914070e00 msgr2=0x7f7914071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 --2- 172.21.15.136:0/2693435324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7914070e00 0x7f7914071210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f79140c6d70 tx=0x7f790401c520 comp rx=0 tx=0).stop 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 -- 172.21.15.136:0/2693435324 shutdown_connections 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 --2- 172.21.15.136:0/2693435324 >> v2:172.21.15.157:3300/0 conn(0x7f79140c7cd0 0x7f79140c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 --2- 172.21.15.136:0/2693435324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7914070e00 0x7f7914071210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 -- 172.21.15.136:0/2693435324 >> 172.21.15.136:0/2693435324 conn(0x7f79140ce320 msgr2=0x7f79140ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:57.947 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 -- 172.21.15.136:0/2693435324 shutdown_connections 2024-04-25T11:19:57.948 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.946+0000 7f791c492640 1 -- 172.21.15.136:0/2693435324 wait complete. 2024-04-25T11:19:57.949 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 Processor -- start 2024-04-25T11:19:57.949 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- start start 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 0x7f7914145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 0x7f7914140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79140c8ca0 0x7f79141410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f79140c6170 con 0x7f79140c7cd0 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7914165fb0 con 0x7f7914070e00 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79141f4410 con 0x7f79140c8ca0 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f7919a2d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 0x7f7914140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f7919a2d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 0x7f7914140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49566/0 (socket says 172.21.15.136:49566) 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f7919a2d640 1 -- 172.21.15.136:0/2628582702 learned_addr learned my addr 172.21.15.136:0/2628582702 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 0x7f7914145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:57.952 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791aa2f640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79140c8ca0 0x7f79141410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 -- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79140c8ca0 msgr2=0x7f79141410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79140c8ca0 0x7f79141410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 -- 172.21.15.136:0/2628582702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 msgr2=0x7f7914140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 0x7f7914140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 -- 172.21.15.136:0/2628582702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7914146030 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791a22e640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 0x7f7914145af0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f791000c390 tx=0x7f791000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79100052d0 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7910005470 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79141f4660 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910007260 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79141452f0 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7910007400 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.950+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f79100039c0 con 0x7f7914070e00 2024-04-25T11:19:57.953 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.954+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7914050750 con 0x7f7914070e00 2024-04-25T11:19:57.955 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:57.954+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7910003ca0 con 0x7f7914070e00 2024-04-25T11:19:58.010 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.010+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79140c73d0 con 0x7f7914070e00 2024-04-25T11:19:58.012 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.012+0000 7feb9a04c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:19:58.012 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.010+0000 7f790affd640 1 -- 172.21.15.136:0/2628582702 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f79100067e0 con 0x7f7914070e00 2024-04-25T11:19:58.012 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-25T11:19:58.012 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":10,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:19:56.659894+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 msgr2=0x7f7914145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 0x7f7914145af0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f791000c390 tx=0x7f791000c850 comp rx=0 tx=0).stop 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 shutdown_connections 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79140c8ca0 0x7f79141410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79140c7cd0 0x7f7914140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:58.013 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 --2- 172.21.15.136:0/2628582702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7914070e00 0x7f7914145af0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:19:58.014 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 >> 172.21.15.136:0/2628582702 conn(0x7f79140ce320 msgr2=0x7f791406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:19:58.014 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 shutdown_connections 2024-04-25T11:19:58.014 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:19:58.014+0000 7f791c492640 1 -- 172.21.15.136:0/2628582702 wait complete. 2024-04-25T11:19:58.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T11:19:58.088 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.088+0000 7feb9a04c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:19:58.270 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.272+0000 7feb9a04c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:19:58.297 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.298+0000 7f9cdc3c1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T11:19:58.344 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.344+0000 7feb9a04c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:19:58.373 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.374+0000 7f9cdc3c1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T11:19:58.553 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.554+0000 7f9cdc3c1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T11:19:58.627 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.626+0000 7f9cdc3c1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T11:19:58.637 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.636+0000 7feb9a04c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:19:58.698 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:58.700+0000 7feb9a04c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:19:58.924 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.926+0000 7f9cdc3c1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T11:19:58.986 INFO:tasks.ceph.mgr.y.smithi136.stderr:2024-04-25T11:19:58.986+0000 7f9cdc3c1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T11:19:59.757 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:59.756+0000 7feb40b00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T11:19:59.757 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-04-25T11:19:59.756+0000 7feb40b00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T11:20:01.123 INFO:tasks.ceph.osd.0.smithi136.stderr:2024-04-25T11:20:01.122+0000 7eff528037c0 -1 osd.0 0 log_to_monitors true 2024-04-25T11:20:01.187 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-04-25T11:20:01.188+0000 7faf826d07c0 -1 osd.5 0 log_to_monitors true 2024-04-25T11:20:01.224 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-04-25T11:20:01.224+0000 7feaf20a17c0 -1 osd.4 0 log_to_monitors true 2024-04-25T11:20:01.233 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-04-25T11:20:01.232+0000 7f6ed1b337c0 -1 osd.6 0 log_to_monitors true 2024-04-25T11:20:01.250 INFO:tasks.ceph.osd.3.smithi136.stderr:2024-04-25T11:20:01.250+0000 7fb5152bb7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T11:20:01.258 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-04-25T11:20:01.260+0000 7f1554c257c0 -1 osd.7 0 log_to_monitors true 2024-04-25T11:20:01.334 INFO:tasks.ceph.osd.2.smithi136.stderr:2024-04-25T11:20:01.334+0000 7fd0f7f207c0 -1 osd.2 0 log_to_monitors true 2024-04-25T11:20:01.377 INFO:tasks.ceph.osd.1.smithi136.stderr:2024-04-25T11:20:01.378+0000 7fcd283e17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T11:20:04.451 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-04-25T11:20:04.452+0000 7f6ecdada640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T11:20:04.471 INFO:tasks.ceph.osd.0.smithi136.stderr:2024-04-25T11:20:04.470+0000 7eff4e7aa640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T11:20:04.479 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-04-25T11:20:04.480+0000 7f6ec58db640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.483 INFO:tasks.ceph.osd.1.smithi136.stderr:2024-04-25T11:20:04.482+0000 7fcd24388640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T11:20:04.496 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-04-25T11:20:04.496+0000 7feaee048640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T11:20:04.498 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-04-25T11:20:04.496+0000 7f1550bcc640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T11:20:04.500 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-04-25T11:20:04.500+0000 7faf7e677640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T11:20:04.501 INFO:tasks.ceph.osd.0.smithi136.stderr:2024-04-25T11:20:04.502+0000 7eff465ab640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.515 INFO:tasks.ceph.osd.1.smithi136.stderr:2024-04-25T11:20:04.514+0000 7fcd1c189640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.531 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-04-25T11:20:04.532+0000 7feae5e49640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.531 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-04-25T11:20:04.532+0000 7f15489cd640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.531 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-04-25T11:20:04.532+0000 7faf76478640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.536 INFO:tasks.ceph.osd.3.smithi136.stderr:2024-04-25T11:20:04.538+0000 7fb511262640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T11:20:04.551 INFO:tasks.ceph.osd.2.smithi136.stderr:2024-04-25T11:20:04.550+0000 7fd0f3ec7640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T11:20:04.565 INFO:tasks.ceph.osd.3.smithi136.stderr:2024-04-25T11:20:04.566+0000 7fb509063640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.581 INFO:tasks.ceph.osd.2.smithi136.stderr:2024-04-25T11:20:04.582+0000 7fd0ebcc8640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T11:20:04.843 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:04.984 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.982+0000 7f72f893c640 1 Processor -- start 2024-04-25T11:20:04.984 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.982+0000 7f72f893c640 1 -- start start 2024-04-25T11:20:04.986 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f893c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 0x7f72f40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.986 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f893c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72f40cd9a0 0x7f72f4164850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f893c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f40c8840 con 0x7f72f40c4ae0 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f893c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f72f40c8ae0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f893c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f72f40c8d80 con 0x7f72f415a950 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72f40cd9a0 0x7f72f4164850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72f40cd9a0 0x7f72f4164850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35936/0 (socket says 172.21.15.136:35936) 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 -- 172.21.15.136:0/2622635675 learned_addr learned my addr 172.21.15.136:0/2622635675 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f1d9b640 1 --2- 172.21.15.136:0/2622635675 >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 0x7f72f40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 -- 172.21.15.136:0/2622635675 >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 msgr2=0x7f72f40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 --2- 172.21.15.136:0/2622635675 >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 0x7f72f40cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 -- 172.21.15.136:0/2622635675 >> v1:172.21.15.157:6789/0 conn(0x7f72f40c4ae0 legacy=0x7f72f4154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f2d9d640 1 -- 172.21.15.136:0/2622635675 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f40c4ef0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.987 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.986+0000 7f72f1d9b640 1 --2- 172.21.15.136:0/2622635675 >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 0x7f72f40cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:04.988 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f2d9d640 1 --2- 172.21.15.136:0/2622635675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f4164850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72f40718f0 tx=0x7f72ec01c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd937aede48f46b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:04.988 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f159a640 1 -- 172.21.15.136:0/2622635675 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0a5670 con 0x7f72f40cd9a0 2024-04-25T11:20:04.988 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f159a640 1 -- 172.21.15.136:0/2622635675 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72ec0a5810 con 0x7f72f40cd9a0 2024-04-25T11:20:04.988 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f159a640 1 -- 172.21.15.136:0/2622635675 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0a5be0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.989 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- 172.21.15.136:0/2622635675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 msgr2=0x7f72f4164850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:04.989 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 --2- 172.21.15.136:0/2622635675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f4164850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f72f40718f0 tx=0x7f72ec01c9b0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.989 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- 172.21.15.136:0/2622635675 shutdown_connections 2024-04-25T11:20:04.989 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 --2- 172.21.15.136:0/2622635675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f4164850 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.990 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 --2- 172.21.15.136:0/2622635675 >> v2:172.21.15.157:3300/0 conn(0x7f72f415a950 0x7f72f40cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.990 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- 172.21.15.136:0/2622635675 >> 172.21.15.136:0/2622635675 conn(0x7f72f4070780 msgr2=0x7f72f4070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:04.990 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- 172.21.15.136:0/2622635675 shutdown_connections 2024-04-25T11:20:04.990 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- 172.21.15.136:0/2622635675 wait complete. 2024-04-25T11:20:04.991 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 Processor -- start 2024-04-25T11:20:04.991 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.990+0000 7f72f893c640 1 -- start start 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72f40c4ae0 0x7f72f4143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 0x7f72f4073c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f72f40c91f0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f72f415ad60 con 0x7f72f40c4ae0 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72f40c8420 con 0x7f72f415a950 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.994 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35950/0 (socket says 172.21.15.136:35950) 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 -- 172.21.15.136:0/1401255049 learned_addr learned my addr 172.21.15.136:0/1401255049 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f259c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72f40c4ae0 0x7f72f4143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f2d9d640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 0x7f72f4073c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 -- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 msgr2=0x7f72f4073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 0x7f72f4073c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 -- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72f40c4ae0 msgr2=0x7f72f4143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72f40c4ae0 0x7f72f4143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 -- 172.21.15.136:0/1401255049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f414c8b0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f2d9d640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 0x7f72f4073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f1d9b640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c850 tx=0x7f72e800cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8015070 con 0x7f72f40cd9a0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e80077d0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.995 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f4144730 con 0x7f72f40cd9a0 2024-04-25T11:20:04.996 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8007ba0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.996 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f41449d0 con 0x7f72f40cd9a0 2024-04-25T11:20:04.996 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72f4050750 con 0x7f72f40cd9a0 2024-04-25T11:20:04.996 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 4) ==== 93151+0+0 (secure 0 0 0) 0x7f72e8018070 con 0x7f72f40cd9a0 2024-04-25T11:20:04.997 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 0x7f72d40718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:04.997 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f72e808e120 con 0x7f72f40cd9a0 2024-04-25T11:20:04.997 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.994+0000 7f72f259c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 0x7f72d40718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:04.997 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:04.998+0000 7f72f259c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 0x7f72d40718b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f72e40098e0 tx=0x7f72e4009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:05.000 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.002+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f72e805b8c0 con 0x7f72f40cd9a0 2024-04-25T11:20:05.149 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.150+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72f40c8fd0 con 0x7f72f40cd9a0 2024-04-25T11:20:05.151 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.150+0000 7f72e2ffd640 1 -- 172.21.15.136:0/1401255049 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f72e80620e0 con 0x7f72f40cd9a0 2024-04-25T11:20:05.151 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-25T11:20:05.151 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":12,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:02.801741+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:05.153 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 msgr2=0x7f72d40718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:05.153 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 0x7f72d40718b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f72e40098e0 tx=0x7f72e4009060 comp rx=0 tx=0).stop 2024-04-25T11:20:05.153 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 msgr2=0x7f72f414c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:05.153 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c850 tx=0x7f72e800cd10 comp rx=0 tx=0).stop 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 shutdown_connections 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72d406f3a0 0x7f72d40718b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72f415a950 0x7f72f4073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f40cd9a0 0x7f72f414c370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 --2- 172.21.15.136:0/1401255049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72f40c4ae0 0x7f72f4143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 >> 172.21.15.136:0/1401255049 conn(0x7f72f4070780 msgr2=0x7f72f4144f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 shutdown_connections 2024-04-25T11:20:05.154 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:05.154+0000 7f72f893c640 1 -- 172.21.15.136:0/1401255049 wait complete. 2024-04-25T11:20:05.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T11:20:11.976 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:12.128 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.126+0000 7fc5ca43c640 1 Processor -- start 2024-04-25T11:20:12.128 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.126+0000 7fc5ca43c640 1 -- start start 2024-04-25T11:20:12.130 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40c4ae0 0x7fc5c415a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc5c4154010 0x7fc5c40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 0x7fc5c415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c40c8840 con 0x7fc5c4154010 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c40c8ae0 con 0x7fc5c40cd9a0 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5ca43c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c40c8d80 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc5c4154010 0x7fc5c40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc5c4154010 0x7fc5c40cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35962/0 (socket says 172.21.15.136:35962) 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 -- 172.21.15.136:0/3016804396 learned_addr learned my addr 172.21.15.136:0/3016804396 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c3fff640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40c4ae0 0x7fc5c415a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c89d9640 1 --2- 172.21.15.136:0/3016804396 >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 0x7fc5c415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 -- 172.21.15.136:0/3016804396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40c4ae0 msgr2=0x7fc5c415a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40c4ae0 0x7fc5c415a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 -- 172.21.15.136:0/3016804396 >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 msgr2=0x7fc5c415d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.131 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/3016804396 >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 0x7fc5c415d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.132 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c37fe640 1 -- 172.21.15.136:0/3016804396 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c40c4ef0 con 0x7fc5c4154010 2024-04-25T11:20:12.132 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.130+0000 7fc5c89d9640 1 --2- 172.21.15.136:0/3016804396 >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 0x7fc5c415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:12.132 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c4154010 0x7fc5c40cd460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4075050 tx=0x7fc5b401c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29bbe72f37cfcb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.132 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5c2ffd640 1 -- 172.21.15.136:0/3016804396 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40b0070 con 0x7fc5c4154010 2024-04-25T11:20:12.133 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5c2ffd640 1 -- 172.21.15.136:0/3016804396 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b40ab600 con 0x7fc5c4154010 2024-04-25T11:20:12.133 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5c2ffd640 1 -- 172.21.15.136:0/3016804396 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40ab9d0 con 0x7fc5c4154010 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- 172.21.15.136:0/3016804396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c4154010 msgr2=0x7fc5c40cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c4154010 0x7fc5c40cd460 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4075050 tx=0x7fc5b401c1d0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- 172.21.15.136:0/3016804396 shutdown_connections 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c4154010 0x7fc5c40cd460 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/3016804396 >> v2:172.21.15.157:3300/0 conn(0x7fc5c40cd9a0 0x7fc5c415d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/3016804396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40c4ae0 0x7fc5c415a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- 172.21.15.136:0/3016804396 >> 172.21.15.136:0/3016804396 conn(0x7fc5c4070780 msgr2=0x7fc5c4070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- 172.21.15.136:0/3016804396 shutdown_connections 2024-04-25T11:20:12.134 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- 172.21.15.136:0/3016804396 wait complete. 2024-04-25T11:20:12.135 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 Processor -- start 2024-04-25T11:20:12.136 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.134+0000 7fc5ca43c640 1 -- start start 2024-04-25T11:20:12.138 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.138 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 0x7fc5c41f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 0x7fc5c413ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c40c8420 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c40c91f0 con 0x7fc5c4154010 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c40c85a0 con 0x7fc5c40cd9a0 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35966/0 (socket says 172.21.15.136:35966) 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 -- 172.21.15.136:0/2850385252 learned_addr learned my addr 172.21.15.136:0/2850385252 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 0x7fc5c41f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c89d9640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 0x7fc5c413ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 -- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 msgr2=0x7fc5c41f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 0x7fc5c41f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 -- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 msgr2=0x7fc5c413ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 0x7fc5c413ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.139 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 -- 172.21.15.136:0/2850385252 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c415b880 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 0x7fc5c41f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c89d9640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 0x7fc5c413ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c3fff640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e870 tx=0x7fc5b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8017070 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b800ee60 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c415d030 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800cb80 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c41f7cf0 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.140 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c415af90 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.141 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc5b801a070 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.141 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 0x7fc5b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.141 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fc5b808f540 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.141 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.138+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 0x7fc5b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.141 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.142+0000 7fc5c37fe640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 0x7fc5b0071910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40ad000 tx=0x7fc5b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.144 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.146+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc5c415af90 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.304 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5c4050750 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.305 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5c0ff9640 1 -- 172.21.15.136:0/2850385252 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7fc5b805c780 con 0x7fc5c40c4ae0 2024-04-25T11:20:12.306 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-25T11:20:12.306 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":16,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:08.866005+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:12.308 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 msgr2=0x7fc5b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.308 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 0x7fc5b0071910 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40ad000 tx=0x7fc5b4008040 comp rx=0 tx=0).stop 2024-04-25T11:20:12.308 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 msgr2=0x7fc5c415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.308 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e870 tx=0x7fc5b800ed30 comp rx=0 tx=0).stop 2024-04-25T11:20:12.308 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 shutdown_connections 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5c4154010 0x7fc5c413ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc5b006f400 0x7fc5b0071910 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc5c40cd9a0 0x7fc5c41f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 --2- 172.21.15.136:0/2850385252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5c40c4ae0 0x7fc5c415b340 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 >> 172.21.15.136:0/2850385252 conn(0x7fc5c4070780 msgr2=0x7fc5c40c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 shutdown_connections 2024-04-25T11:20:12.309 INFO:teuthology.misc.health.smithi136.stderr:2024-04-25T11:20:12.306+0000 7fc5ca43c640 1 -- 172.21.15.136:0/2850385252 wait complete. 2024-04-25T11:20:12.325 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T11:20:12.325 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T11:20:12.325 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T11:20:12.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T11:20:12.328 INFO:tasks.ceph.mds.a:Started 2024-04-25T11:20:12.328 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T11:20:12.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T11:20:12.330 INFO:tasks.ceph.mds.c:Started 2024-04-25T11:20:12.330 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-25T11:20:12.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-25T11:20:12.332 INFO:tasks.ceph.mds.e:Started 2024-04-25T11:20:12.332 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T11:20:12.333 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T11:20:12.335 INFO:tasks.ceph.mds.b:Started 2024-04-25T11:20:12.335 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T11:20:12.336 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T11:20:12.338 INFO:tasks.ceph.mds.d:Started 2024-04-25T11:20:12.338 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-25T11:20:12.338 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-25T11:20:12.340 INFO:tasks.ceph.mds.f:Started 2024-04-25T11:20:12.341 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T11:20:12.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:12.424 INFO:tasks.ceph.mds.a.smithi136.stdout:starting mds.a at 2024-04-25T11:20:12.425 INFO:tasks.ceph.mds.c.smithi136.stdout:starting mds.c at 2024-04-25T11:20:12.429 INFO:tasks.ceph.mds.e.smithi136.stdout:starting mds.e at 2024-04-25T11:20:12.445 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:20:12.445 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-04-25T11:20:12.446 INFO:tasks.ceph.mds.f.smithi157.stdout:starting mds.f at 2024-04-25T11:20:12.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.478+0000 7fbdae675640 1 Processor -- start 2024-04-25T11:20:12.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.478+0000 7fbdae675640 1 -- start start 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbda80b5060 0x7fbda80c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbda815a430 0x7fbda815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fbda815d750 con 0x7fbda815a430 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbda815da30 con 0x7fbda80b5060 2024-04-25T11:20:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdae675640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbda815dd10 con 0x7fbda80ca290 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbda815a430 0x7fbda815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbda815a430 0x7fbda815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36040/0 (socket says 172.21.15.136:36040) 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 -- 172.21.15.136:0/1059031987 learned_addr learned my addr 172.21.15.136:0/1059031987 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbda77fe640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbda7fff640 1 --2- 172.21.15.136:0/1059031987 >> v2:172.21.15.157:3300/0 conn(0x7fbda80b5060 0x7fbda80c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 -- 172.21.15.136:0/1059031987 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 msgr2=0x7fbda80ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda80ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 -- 172.21.15.136:0/1059031987 >> v2:172.21.15.157:3300/0 conn(0x7fbda80b5060 msgr2=0x7fbda80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 --2- 172.21.15.136:0/1059031987 >> v2:172.21.15.157:3300/0 conn(0x7fbda80b5060 0x7fbda80c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 -- 172.21.15.136:0/1059031987 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda815dff0 con 0x7fbda815a430 2024-04-25T11:20:12.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbdacc12640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda815a430 0x7fbda815a840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbda8068ff0 tx=0x7fbda001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f31f3b33426c50b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbda6ffd640 1 -- 172.21.15.136:0/1059031987 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda00b4070 con 0x7fbda815a430 2024-04-25T11:20:12.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbda6ffd640 1 -- 172.21.15.136:0/1059031987 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda00af400 con 0x7fbda815a430 2024-04-25T11:20:12.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.482+0000 7fbda6ffd640 1 -- 172.21.15.136:0/1059031987 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda00af7d0 con 0x7fbda815a430 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- 172.21.15.136:0/1059031987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda815a430 msgr2=0x7fbda815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda815a430 0x7fbda815a840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbda8068ff0 tx=0x7fbda001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- 172.21.15.136:0/1059031987 shutdown_connections 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda815a430 0x7fbda815a840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- 172.21.15.136:0/1059031987 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda80ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- 172.21.15.136:0/1059031987 >> v2:172.21.15.157:3300/0 conn(0x7fbda80b5060 0x7fbda80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- 172.21.15.136:0/1059031987 >> 172.21.15.136:0/1059031987 conn(0x7fbda8070f70 msgr2=0x7fbda8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- 172.21.15.136:0/1059031987 shutdown_connections 2024-04-25T11:20:12.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- 172.21.15.136:0/1059031987 wait complete. 2024-04-25T11:20:12.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 Processor -- start 2024-04-25T11:20:12.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- start start 2024-04-25T11:20:12.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda815afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbda815a430 0x7fbda8147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbda815ad80 con 0x7fbda80b5060 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8159f10 con 0x7fbda815a430 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdae675640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbda815d3f0 con 0x7fbda80ca290 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36044/0 (socket says 172.21.15.136:36044) 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 -- 172.21.15.136:0/4190885943 learned_addr learned my addr 172.21.15.136:0/4190885943 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda77fe640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda815afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbdacc12640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbda815a430 0x7fbda8147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 -- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 msgr2=0x7fbda815afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda815afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 -- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbda815a430 msgr2=0x7fbda8147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbda815a430 0x7fbda8147c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.486+0000 7fbda7fff640 1 -- 172.21.15.136:0/4190885943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda8153b70 con 0x7fbda80b5060 2024-04-25T11:20:12.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda7fff640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800e870 tx=0x7fbd9800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd98017070 con 0x7fbda80b5060 2024-04-25T11:20:12.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd9800ee60 con 0x7fbda80b5060 2024-04-25T11:20:12.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9800cb20 con 0x7fbda80b5060 2024-04-25T11:20:12.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda81f0ae0 con 0x7fbda80b5060 2024-04-25T11:20:12.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda81f10a0 con 0x7fbda80b5060 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda806a350 con 0x7fbda80b5060 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbd9801a070 con 0x7fbda80b5060 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 0x7fbd94071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbd9808ec70 con 0x7fbda80b5060 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda77fe640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 0x7fbd94071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.490+0000 7fbda77fe640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 0x7fbd94071940 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbda806aab0 tx=0x7fbd9c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.494+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbd9805b860 con 0x7fbda80b5060 2024-04-25T11:20:12.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.646+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbda80cadd0 con 0x7fbda80b5060 2024-04-25T11:20:12.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.646+0000 7fbda4ff9640 1 -- 172.21.15.136:0/4190885943 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7fbd9805fb60 con 0x7fbda80b5060 2024-04-25T11:20:12.648 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:12.648 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":16,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:08.866005+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 msgr2=0x7fbd94071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 0x7fbd94071940 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbda806aab0 tx=0x7fbd9c009190 comp rx=0 tx=0).stop 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 msgr2=0x7fbda81533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd9800e870 tx=0x7fbd9800ed30 comp rx=0 tx=0).stop 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 shutdown_connections 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbda815a430 0x7fbda8147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd9406f430 0x7fbd94071940 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbda80ca290 0x7fbda815afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 --2- 172.21.15.136:0/4190885943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbda80b5060 0x7fbda81533e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 >> 172.21.15.136:0/4190885943 conn(0x7fbda8070f70 msgr2=0x7fbda814a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 shutdown_connections 2024-04-25T11:20:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.650+0000 7fbdae675640 1 -- 172.21.15.136:0/4190885943 wait complete. 2024-04-25T11:20:12.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:20:12.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:20:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 Processor -- start 2024-04-25T11:20:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 -- start start 2024-04-25T11:20:12.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe42806e1b0 0x7fe42815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 0x7fe4280ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe42806e5c0 con 0x7fe42806d780 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe4280c55a0 con 0x7fe42806e1b0 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42cb9e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe42815c730 con 0x7fe4280ca290 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe425d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe42806d780 0x7fe4281595f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52640/0 (socket says 172.21.15.136:52640) 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe425d9b640 1 -- 172.21.15.136:0/1543918230 learned_addr learned my addr 172.21.15.136:0/1543918230 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe426d9d640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 0x7fe4280ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42659c640 1 --2- 172.21.15.136:0/1543918230 >> v2:172.21.15.157:3300/0 conn(0x7fe42806e1b0 0x7fe42815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993410641 0 0) 0x7fe42806e5c0 con 0x7fe42806d780 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe410003530 con 0x7fe42806d780 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3750476839 0 0) 0x7fe410003530 con 0x7fe42806d780 2024-04-25T11:20:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 msgr2=0x7fe4280ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 0x7fe4280ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 >> v2:172.21.15.157:3300/0 conn(0x7fe42806e1b0 msgr2=0x7fe42815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 --2- 172.21.15.136:0/1543918230 >> v2:172.21.15.157:3300/0 conn(0x7fe42806e1b0 0x7fe42815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42815c100 con 0x7fe42806d780 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fe41c0016a0 con 0x7fe42806d780 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 >> v1:172.21.15.136:6789/0 conn(0x7fe42806d780 legacy=0x7fe4281595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 0x7fe4100039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe410003f40 0x7fe410004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410004d50 0x7fe410005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe4280c55a0 con 0x7fe410003f40 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe42815c730 con 0x7fe410004d50 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe41000bad0 con 0x7fe4280ca290 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.814+0000 7fe425d9b640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe410003f40 0x7fe410004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe426d9d640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410004d50 0x7fe410005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 -- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 msgr2=0x7fe4100039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280ca290 0x7fe4100039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 -- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410004d50 msgr2=0x7fe410005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:12.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410004d50 0x7fe410005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 -- 172.21.15.136:0/1543918230 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe410003530 con 0x7fe410003f40 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe425d9b640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe410003f40 0x7fe410004810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c210 tx=0x7fe41c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c589bf4596b535b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c007d80 con 0x7fe410003f40 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41c0a9070 con 0x7fe410003f40 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42559a640 1 -- 172.21.15.136:0/1543918230 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c005d40 con 0x7fe410003f40 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 -- 172.21.15.136:0/1543918230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe410003f40 msgr2=0x7fe410004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/1543918230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe410003f40 0x7fe410004810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00c210 tx=0x7fe41c01cca0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe426d9d640 1 -- 172.21.15.136:0/1543918230 reap_dead start 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 -- 172.21.15.136:0/1543918230 shutdown_connections 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 -- 172.21.15.136:0/1543918230 >> 172.21.15.136:0/1543918230 conn(0x7fe428070f70 msgr2=0x7fe428068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 -- 172.21.15.136:0/1543918230 shutdown_connections 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.818+0000 7fe42cb9e640 1 -- 172.21.15.136:0/1543918230 wait complete. 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 Processor -- start 2024-04-25T11:20:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- start start 2024-04-25T11:20:12.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 0x7fe4281f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280cd3b0 0x7fe4280cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe42806db90 con 0x7fe428149ac0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe42815c3c0 con 0x7fe410003f40 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4280caab0 con 0x7fe4280cd3b0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36068/0 (socket says 172.21.15.136:36068) 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 -- 172.21.15.136:0/74032908 learned_addr learned my addr 172.21.15.136:0/74032908 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe425d9b640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280cd3b0 0x7fe4280cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42659c640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 0x7fe4281f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 -- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280cd3b0 msgr2=0x7fe4280cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280cd3b0 0x7fe4280cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 -- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 msgr2=0x7fe4281f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 0x7fe4281f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 -- 172.21.15.136:0/74032908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42815bdf0 con 0x7fe428149ac0 2024-04-25T11:20:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42659c640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 0x7fe4281f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe426d9d640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe428076910 tx=0x7fe41800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41801c070 con 0x7fe428149ac0 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe418017400 con 0x7fe428149ac0 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42815aaf0 con 0x7fe428149ac0 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4180177f0 con 0x7fe428149ac0 2024-04-25T11:20:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42814a410 con 0x7fe428149ac0 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.822+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4280cdc90 con 0x7fe428149ac0 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.826+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe41801f070 con 0x7fe428149ac0 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.826+0000 7fe416ffd640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 0x7fe408071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.826+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe41808f3c0 con 0x7fe428149ac0 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.826+0000 7fe42659c640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 0x7fe408071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.826+0000 7fe42659c640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 0x7fe408071910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe4200047c0 tx=0x7fe420008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:12.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.830+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe418012070 con 0x7fe428149ac0 2024-04-25T11:20:12.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.978+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe42806a350 con 0x7fe428149ac0 2024-04-25T11:20:12.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe416ffd640 1 -- 172.21.15.136:0/74032908 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fe41805c770 con 0x7fe428149ac0 2024-04-25T11:20:12.981 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 msgr2=0x7fe408071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 0x7fe408071910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe4200047c0 tx=0x7fe420008040 comp rx=0 tx=0).stop 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 msgr2=0x7fe428149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe428076910 tx=0x7fe41800f4a0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 shutdown_connections 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe428149ac0 0x7fe428149ed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe40806f400 0x7fe408071910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4280cd3b0 0x7fe4280cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 --2- 172.21.15.136:0/74032908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe410003f40 0x7fe4281f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:12.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 >> 172.21.15.136:0/74032908 conn(0x7fe428070f70 msgr2=0x7fe42815b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:12.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 shutdown_connections 2024-04-25T11:20:12.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:12.982+0000 7fe42cb9e640 1 -- 172.21.15.136:0/74032908 wait complete. 2024-04-25T11:20:13.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.134+0000 7f8b936b7640 1 Processor -- start 2024-04-25T11:20:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.134+0000 7f8b936b7640 1 -- start start 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b936b7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b8c153190 0x7f8b8c0c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b936b7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 0x7f8b8c158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b936b7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b8c1596f0 con 0x7f8b8c1592e0 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b936b7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8c0c8370 con 0x7f8b8c153190 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b936b7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8c0c8090 con 0x7f8b8c0c9570 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b8c153190 0x7f8b8c0c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b8c153190 0x7f8b8c0c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36072/0 (socket says 172.21.15.136:36072) 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 -- 172.21.15.136:0/1253850791 learned_addr learned my addr 172.21.15.136:0/1253850791 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b90c52640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 0x7f8b8c158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 -- 172.21.15.136:0/1253850791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 msgr2=0x7f8b8c158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 0x7f8b8c158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 -- 172.21.15.136:0/1253850791 >> v1:172.21.15.157:6789/0 conn(0x7f8b8c1592e0 legacy=0x7f8b8c15c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b91453640 1 -- 172.21.15.136:0/1253850791 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c0c6aa0 con 0x7f8b8c153190 2024-04-25T11:20:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.138+0000 7f8b90c52640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 0x7f8b8c158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b91453640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c153190 0x7f8b8c0c9030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400d150 tx=0x7f8b840213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989c4245931720ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b7ffff640 1 -- 172.21.15.136:0/1253850791 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b840b6070 con 0x7f8b8c153190 2024-04-25T11:20:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b7ffff640 1 -- 172.21.15.136:0/1253850791 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8400b2d0 con 0x7f8b8c153190 2024-04-25T11:20:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b7ffff640 1 -- 172.21.15.136:0/1253850791 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400b480 con 0x7f8b8c153190 2024-04-25T11:20:13.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- 172.21.15.136:0/1253850791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c153190 msgr2=0x7f8b8c0c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:13.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c153190 0x7f8b8c0c9030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400d150 tx=0x7f8b840213e0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- 172.21.15.136:0/1253850791 shutdown_connections 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c0c9570 0x7f8b8c158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 --2- 172.21.15.136:0/1253850791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c153190 0x7f8b8c0c9030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- 172.21.15.136:0/1253850791 >> 172.21.15.136:0/1253850791 conn(0x7f8b8c0c9d50 msgr2=0x7f8b8c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- 172.21.15.136:0/1253850791 shutdown_connections 2024-04-25T11:20:13.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- 172.21.15.136:0/1253850791 wait complete. 2024-04-25T11:20:13.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 Processor -- start 2024-04-25T11:20:13.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.142+0000 7f8b936b7640 1 -- start start 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 0x7f8b8c1ee880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c1592e0 0x7f8b8c14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c0c7340 con 0x7f8b8c0c9570 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c0c7060 con 0x7f8b8c153190 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8c0ca870 con 0x7f8b8c1592e0 2024-04-25T11:20:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36074/0 (socket says 172.21.15.136:36074) 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 -- 172.21.15.136:0/873685739 learned_addr learned my addr 172.21.15.136:0/873685739 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91c54640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c1592e0 0x7f8b8c14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 -- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c1592e0 msgr2=0x7f8b8c14e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c1592e0 0x7f8b8c14e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b90c52640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 0x7f8b8c1ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 -- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 msgr2=0x7f8b8c1ee880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 0x7f8b8c1ee880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 -- 172.21.15.136:0/873685739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c1eedc0 con 0x7f8b8c0c9570 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b90c52640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 0x7f8b8c1ee880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b91453640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b8401fa90 tx=0x7f8b8401e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b840b6040 con 0x7f8b8c0c9570 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b840c2070 con 0x7f8b8c0c9570 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400b480 con 0x7f8b8c0c9570 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8c14cc10 con 0x7f8b8c0c9570 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8c14f180 con 0x7f8b8c0c9570 2024-04-25T11:20:13.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8c06a350 con 0x7f8b8c0c9570 2024-04-25T11:20:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8b840bc640 con 0x7f8b8c0c9570 2024-04-25T11:20:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 0x7f8b70071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8b8412f210 con 0x7f8b8c0c9570 2024-04-25T11:20:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.146+0000 7f8b90c52640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 0x7f8b70071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.150+0000 7f8b90c52640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 0x7f8b70071940 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c15d1a0 tx=0x7f8b80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:13.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.150+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8b8401aeb0 con 0x7f8b8c0c9570 2024-04-25T11:20:13.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.330+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8c0c86c0 con 0x7f8b8c0c9570 2024-04-25T11:20:13.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b7dffb640 1 -- 172.21.15.136:0/873685739 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f8b840fc570 con 0x7f8b8c0c9570 2024-04-25T11:20:13.333 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:13.333 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T11:20:13.333 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 2 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 msgr2=0x7f8b70071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 0x7f8b70071940 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c15d1a0 tx=0x7f8b80008040 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 msgr2=0x7f8b8c15ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b8401fa90 tx=0x7f8b8401e180 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 shutdown_connections 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8b7006f430 0x7f8b70071940 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b8c1592e0 0x7f8b8c14e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b8c153190 0x7f8b8c1ee880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 --2- 172.21.15.136:0/873685739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b8c0c9570 0x7f8b8c15ac90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 >> 172.21.15.136:0/873685739 conn(0x7f8b8c0c9d50 msgr2=0x7f8b8c146d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:13.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 shutdown_connections 2024-04-25T11:20:13.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:13.334+0000 7f8b936b7640 1 -- 172.21.15.136:0/873685739 wait complete. 2024-04-25T11:20:15.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:15.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 Processor -- start 2024-04-25T11:20:15.513 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- start start 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd24c0c8cd0 0x7fd24c14f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 0x7fd24c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd24c0cbad0 con 0x7fd24c0ca010 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd24c0c8550 con 0x7fd24c0c8cd0 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c0c7c20 con 0x7fd24c0ce260 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24ad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd24c0ca010 0x7fd24c0ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41608/0 (socket says 172.21.15.136:41608) 2024-04-25T11:20:15.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24ad9d640 1 -- 172.21.15.136:0/654113117 learned_addr learned my addr 172.21.15.136:0/654113117 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 --2- 172.21.15.136:0/654113117 >> v2:172.21.15.157:3300/0 conn(0x7fd24c0c8cd0 0x7fd24c14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd249d9b640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 0x7fd24c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3066329616 0 0) 0x7fd24c0cbad0 con 0x7fd24c0ca010 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd234003530 con 0x7fd24c0ca010 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 43150410 0 0) 0x7fd234003530 con 0x7fd24c0ca010 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 msgr2=0x7fd24c0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 0x7fd24c0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 >> v2:172.21.15.157:3300/0 conn(0x7fd24c0c8cd0 msgr2=0x7fd24c14f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 --2- 172.21.15.136:0/654113117 >> v2:172.21.15.157:3300/0 conn(0x7fd24c0c8cd0 0x7fd24c14f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c0c7f00 con 0x7fd24c0ca010 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd244001a00 con 0x7fd24c0ca010 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 >> v1:172.21.15.136:6789/0 conn(0x7fd24c0ca010 legacy=0x7fd24c0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2340039c0 0x7fd234003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd234004810 0x7fd234004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 0x7fd234005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c0c8550 con 0x7fd2340039c0 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c0c7c20 con 0x7fd234004810 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd23400ba50 con 0x7fd24c0ce260 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2340039c0 0x7fd234003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd249d9b640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd234004810 0x7fd234004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 -- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 msgr2=0x7fd234005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c0ce260 0x7fd234005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 -- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd234004810 msgr2=0x7fd234004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd234004810 0x7fd234004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 -- 172.21.15.136:0/654113117 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd234003530 con 0x7fd2340039c0 2024-04-25T11:20:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd249d9b640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd234004810 0x7fd234004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24a59c640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2340039c0 0x7fd234003e50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009ae0 tx=0x7fd23c01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69d7baaffd19ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0b2070 con 0x7fd2340039c0 2024-04-25T11:20:15.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23c0a5740 con 0x7fd2340039c0 2024-04-25T11:20:15.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24959a640 1 -- 172.21.15.136:0/654113117 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0096b0 con 0x7fd2340039c0 2024-04-25T11:20:15.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- 172.21.15.136:0/654113117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2340039c0 msgr2=0x7fd234003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 --2- 172.21.15.136:0/654113117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2340039c0 0x7fd234003e50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009ae0 tx=0x7fd23c01cb40 comp rx=0 tx=0).stop 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd24ad9d640 1 -- 172.21.15.136:0/654113117 reap_dead start 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- 172.21.15.136:0/654113117 shutdown_connections 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.517+0000 7fd250ae4640 1 -- 172.21.15.136:0/654113117 >> 172.21.15.136:0/654113117 conn(0x7fd24c06f9d0 msgr2=0x7fd24c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- 172.21.15.136:0/654113117 shutdown_connections 2024-04-25T11:20:15.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- 172.21.15.136:0/654113117 wait complete. 2024-04-25T11:20:15.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 Processor -- start 2024-04-25T11:20:15.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- start start 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 0x7fd24c0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c15bec0 0x7fd24c06d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c0cb720 con 0x7fd24c14a180 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c14fbc0 con 0x7fd2340039c0 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd250ae4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c0cd1a0 con 0x7fd24c15bec0 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46092/0 (socket says 172.21.15.136:46092) 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 -- 172.21.15.136:0/2124184608 learned_addr learned my addr 172.21.15.136:0/2124184608 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24a59c640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 0x7fd24c0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd249d9b640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c15bec0 0x7fd24c06d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 -- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c15bec0 msgr2=0x7fd24c06d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c15bec0 0x7fd24c06d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 -- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 msgr2=0x7fd24c0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 0x7fd24c0cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 -- 172.21.15.136:0/2124184608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c0cccd0 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24a59c640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 0x7fd24c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.521+0000 7fd24ad9d640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd244004c00 tx=0x7fd24400ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24401b070 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24400da20 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24400ddf0 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24c15aae0 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24c15b080 con 0x7fd24c14a180 2024-04-25T11:20:15.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24c06a350 con 0x7fd24c14a180 2024-04-25T11:20:15.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd24401e070 con 0x7fd24c14a180 2024-04-25T11:20:15.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 0x7fd22c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd244091640 con 0x7fd24c14a180 2024-04-25T11:20:15.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd24a59c640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 0x7fd22c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.525+0000 7fd24a59c640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 0x7fd22c071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0af000 tx=0x7fd23c0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.529+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd24405ea80 con 0x7fd24c14a180 2024-04-25T11:20:15.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.717+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd24c156790 con 0x7fd24c14a180 2024-04-25T11:20:15.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.717+0000 7fd23affd640 1 -- 172.21.15.136:0/2124184608 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5521 (secure 0 0 0) 0x7fd24405e380 con 0x7fd24c14a180 2024-04-25T11:20:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4303,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T11:20:15.721 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 3 2024-04-25T11:20:15.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 msgr2=0x7fd22c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 0x7fd22c071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0af000 tx=0x7fd23c0046d0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 msgr2=0x7fd24c148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd244004c00 tx=0x7fd24400ff00 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 shutdown_connections 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd24c14a180 0x7fd24c148890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd22c06f400 0x7fd22c071910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd24c15bec0 0x7fd24c06d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 --2- 172.21.15.136:0/2124184608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2340039c0 0x7fd24c0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 >> 172.21.15.136:0/2124184608 conn(0x7fd24c06f9d0 msgr2=0x7fd24c0ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 shutdown_connections 2024-04-25T11:20:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.721+0000 7fd250ae4640 1 -- 172.21.15.136:0/2124184608 wait complete. 2024-04-25T11:20:15.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:15.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.877+0000 7f75546af640 1 Processor -- start 2024-04-25T11:20:15.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.877+0000 7f75546af640 1 -- start start 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f75546af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f754c143170 0x7f754c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f75546af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754c143580 con 0x7f754c070c80 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f75546af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754c15dd20 con 0x7f754c070870 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f75546af640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f754c15e000 con 0x7f754c143170 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f754c070c80 0x7f754c164790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41616/0 (socket says 172.21.15.136:41616) 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 -- 172.21.15.136:0/4222110796 learned_addr learned my addr 172.21.15.136:0/4222110796 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f755244b640 1 --2- 172.21.15.136:0/4222110796 >> v2:172.21.15.157:3300/0 conn(0x7f754c143170 0x7f754c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124352847 0 0) 0x7f75440014a0 con 0x7f754c070c80 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7538003530 con 0x7f754c070c80 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2799523623 0 0) 0x7f7538003530 con 0x7f754c070c80 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 >> v2:172.21.15.157:3300/0 conn(0x7f754c143170 msgr2=0x7f754c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 --2- 172.21.15.136:0/4222110796 >> v2:172.21.15.157:3300/0 conn(0x7f754c143170 0x7f754c15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 >> v1:172.21.15.157:6789/0 conn(0x7f754c070870 legacy=0x7f754c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c06fd90 con 0x7f754c070c80 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f7544001bd0 con 0x7f754c070c80 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 >> v1:172.21.15.136:6789/0 conn(0x7f754c070c80 legacy=0x7f754c164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380039c0 0x7f7538003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7538004810 0x7f7538004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 0x7f753800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f754c15e000 con 0x7f75380051d0 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f753800c160 con 0x7f7538004810 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f753800c360 con 0x7f75380039c0 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 0x7f753800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7551c4a640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7538004810 0x7f7538004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f755244b640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380039c0 0x7f7538003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 -- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380039c0 msgr2=0x7f7538003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380039c0 0x7f7538003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 -- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7538004810 msgr2=0x7f7538004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:15.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7538004810 0x7f7538004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.881+0000 7f7552c4c640 1 -- 172.21.15.136:0/4222110796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7538003530 con 0x7f75380051d0 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f7552c4c640 1 -- 172.21.15.136:0/4222110796 reap_dead start 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f7552c4c640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 0x7f753800b9d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f754400aa20 tx=0x7f754401d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a853fc11a3f9b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754400a670 con 0x7f75380051d0 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75440ad650 con 0x7f75380051d0 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f7551449640 1 -- 172.21.15.136:0/4222110796 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75440ada40 con 0x7f75380051d0 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 msgr2=0x7f753800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 0x7f753800b9d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f754400aa20 tx=0x7f754401d2a0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- 172.21.15.136:0/4222110796 shutdown_connections 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 --2- 172.21.15.136:0/4222110796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75380051d0 0x7f753800b9d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- 172.21.15.136:0/4222110796 >> 172.21.15.136:0/4222110796 conn(0x7f754c0c9d50 msgr2=0x7f754c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- 172.21.15.136:0/4222110796 shutdown_connections 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- 172.21.15.136:0/4222110796 wait complete. 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 Processor -- start 2024-04-25T11:20:15.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.885+0000 7f75546af640 1 -- start start 2024-04-25T11:20:15.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380051d0 0x7f754c15b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 0x7f754c13ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f754c15e2e0 con 0x7f754c15af30 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f754c070350 con 0x7f754c15d1f0 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f754c0712a0 con 0x7f75380051d0 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46106/0 (socket says 172.21.15.136:46106) 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 -- 172.21.15.136:0/92448450 learned_addr learned my addr 172.21.15.136:0/92448450 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7552c4c640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 0x7f754c13ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f755244b640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380051d0 0x7f754c15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 -- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380051d0 msgr2=0x7f754c15b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380051d0 0x7f754c15b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 -- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 msgr2=0x7f754c13ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 0x7f754c13ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 -- 172.21.15.136:0/92448450 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c15bad0 con 0x7f754c15af30 2024-04-25T11:20:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7552c4c640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 0x7f754c13ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f7551c4a640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f754800e470 tx=0x7f754800e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548019070 con 0x7f754c15af30 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75480052e0 con 0x7f754c15af30 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c1efc50 con 0x7f754c15af30 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548003820 con 0x7f754c15af30 2024-04-25T11:20:15.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1f01d0 con 0x7f754c15af30 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754c06a350 con 0x7f754c15af30 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f754801c070 con 0x7f754c15af30 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 0x7f7530071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.889+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f754808e2b0 con 0x7f754c15af30 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.893+0000 7f755244b640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 0x7f7530071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:15.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.893+0000 7f755244b640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 0x7f7530071910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7540009930 tx=0x7f7540008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:15.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:15.897+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f754805b080 con 0x7f754c15af30 2024-04-25T11:20:16.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.045+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f754c15d760 con 0x7f754c15af30 2024-04-25T11:20:16.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f753effd640 1 -- 172.21.15.136:0/92448450 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7f754805f380 con 0x7f754c15af30 2024-04-25T11:20:16.045 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:16.045 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":16,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:08.866005+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:16.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 msgr2=0x7f7530071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 0x7f7530071910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7540009930 tx=0x7f7540008040 comp rx=0 tx=0).stop 2024-04-25T11:20:16.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 msgr2=0x7f754c13c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f754800e470 tx=0x7f754800e870 comp rx=0 tx=0).stop 2024-04-25T11:20:16.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 shutdown_connections 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f754c15d1f0 0x7f754c13ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c15af30 0x7f754c13c530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f753006f400 0x7f7530071910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 --2- 172.21.15.136:0/92448450 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75380051d0 0x7f754c15b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 >> 172.21.15.136:0/92448450 conn(0x7f754c0c9d50 msgr2=0x7f754c1e3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 shutdown_connections 2024-04-25T11:20:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.049+0000 7f75546af640 1 -- 172.21.15.136:0/92448450 wait complete. 2024-04-25T11:20:16.064 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T11:20:16.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:20:16.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 Processor -- start 2024-04-25T11:20:16.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 -- start start 2024-04-25T11:20:16.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd3440729c0 0x7fd344142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3440739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34415d590 con 0x7fd344072dd0 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd34415d340 con 0x7fd3440729c0 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd34aca6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd34415e1a0 con 0x7fd344073590 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd348a42640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd3440729c0 0x7fd344142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd343fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd344072dd0 0x7fd3440c7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41632/0 (socket says 172.21.15.136:41632) 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.221+0000 7fd343fff640 1 -- 172.21.15.136:0/3473742522 learned_addr learned my addr 172.21.15.136:0/3473742522 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd349243640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3440739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd348a42640 1 -- 172.21.15.136:0/3473742522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 msgr2=0x7fd3440739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd348a42640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3440739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd348a42640 1 -- 172.21.15.136:0/3473742522 >> v1:172.21.15.136:6789/0 conn(0x7fd344072dd0 legacy=0x7fd3440c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd348a42640 1 -- 172.21.15.136:0/3473742522 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34415d870 con 0x7fd3440729c0 2024-04-25T11:20:16.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd348a42640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3440729c0 0x7fd344142cb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3340097c0 tx=0x7fd33401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d67dd1c262bdf8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd3437fe640 1 -- 172.21.15.136:0/3473742522 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3340a7070 con 0x7fd3440729c0 2024-04-25T11:20:16.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd3437fe640 1 -- 172.21.15.136:0/3473742522 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3340b0070 con 0x7fd3440729c0 2024-04-25T11:20:16.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd3437fe640 1 -- 172.21.15.136:0/3473742522 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3340ab890 con 0x7fd3440729c0 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 -- 172.21.15.136:0/3473742522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3440729c0 msgr2=0x7fd344142cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3440729c0 0x7fd344142cb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3340097c0 tx=0x7fd33401c470 comp rx=0 tx=0).stop 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 -- 172.21.15.136:0/3473742522 shutdown_connections 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3440739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 --2- 172.21.15.136:0/3473742522 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3440729c0 0x7fd344142cb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 -- 172.21.15.136:0/3473742522 >> 172.21.15.136:0/3473742522 conn(0x7fd3440c9d50 msgr2=0x7fd3440ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 -- 172.21.15.136:0/3473742522 shutdown_connections 2024-04-25T11:20:16.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.225+0000 7fd34aca6640 1 -- 172.21.15.136:0/3473742522 wait complete. 2024-04-25T11:20:16.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 Processor -- start 2024-04-25T11:20:16.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- start start 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3440729c0 0x7fd3441f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3441f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd3440c76a0 con 0x7fd344072dd0 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd3440c7520 con 0x7fd3440729c0 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd3441431f0 con 0x7fd344073590 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46114/0 (socket says 172.21.15.136:46114) 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 -- 172.21.15.136:0/2629201743 learned_addr learned my addr 172.21.15.136:0/2629201743 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd348a42640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3440729c0 0x7fd3441f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd349243640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3441f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 -- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 msgr2=0x7fd3441f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3441f6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 -- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3440729c0 msgr2=0x7fd3441f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3440729c0 0x7fd3441f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 -- 172.21.15.136:0/2629201743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34413ede0 con 0x7fd344072dd0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd343fff640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c340 tx=0x7fd33800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338017070 con 0x7fd344072dd0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd338005cf0 con 0x7fd344072dd0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380072f0 con 0x7fd344072dd0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd344135e90 con 0x7fd344072dd0 2024-04-25T11:20:16.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.229+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3441f3270 con 0x7fd344072dd0 2024-04-25T11:20:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34406a350 con 0x7fd344072dd0 2024-04-25T11:20:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd33801a070 con 0x7fd344072dd0 2024-04-25T11:20:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd3417fa640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 0x7fd330071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd33808e1f0 con 0x7fd344072dd0 2024-04-25T11:20:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd348a42640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 0x7fd330071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.233+0000 7fd348a42640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 0x7fd330071940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3340af000 tx=0x7fd3340a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.237+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd33805b4c0 con 0x7fd344072dd0 2024-04-25T11:20:16.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.397+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd3440c86d0 con 0x7fd344072dd0 2024-04-25T11:20:16.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.397+0000 7fd3417fa640 1 -- 172.21.15.136:0/2629201743 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fd33805adc0 con 0x7fd344072dd0 2024-04-25T11:20:16.394 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:20:16.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 msgr2=0x7fd330071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 0x7fd330071940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3340af000 tx=0x7fd3340a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 msgr2=0x7fd34413e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c340 tx=0x7fd33800c800 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 shutdown_connections 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd33006f430 0x7fd330071940 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd344073590 0x7fd3441f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd344072dd0 0x7fd34413e650 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 --2- 172.21.15.136:0/2629201743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3440729c0 0x7fd3441f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 >> 172.21.15.136:0/2629201743 conn(0x7fd3440c9d50 msgr2=0x7fd3440ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 shutdown_connections 2024-04-25T11:20:16.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.401+0000 7fd34aca6640 1 -- 172.21.15.136:0/2629201743 wait complete. 2024-04-25T11:20:16.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T11:20:16.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 Processor -- start 2024-04-25T11:20:16.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 -- start start 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58dc167a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58dc15c6f0 con 0x7f58dc15c0d0 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58dc15c990 con 0x7f58dc15ff60 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58e2048640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f58dc0c6310 con 0x7f58dc160370 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58db7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f58dc15c0d0 0x7f58dc0cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41646/0 (socket says 172.21.15.136:41646) 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58db7fe640 1 -- 172.21.15.136:0/3554358993 learned_addr learned my addr 172.21.15.136:0/3554358993 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58dbfff640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58dc167a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819656221 0 0) 0x7f58dc15c6f0 con 0x7f58dc15c0d0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c4003530 con 0x7f58dc15c0d0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 8431069 0 0) 0x7f58c4003530 con 0x7f58dc15c0d0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 msgr2=0x7f58dc167a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58dc167a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 >> v1:172.21.15.157:6789/0 conn(0x7f58dc15ff60 legacy=0x7f58dc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc15cc30 con 0x7f58dc15c0d0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f58d4001a00 con 0x7f58dc15c0d0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 >> v1:172.21.15.136:6789/0 conn(0x7f58dc15c0d0 legacy=0x7f58dc0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.561+0000 7f58da7fc640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c40039f0 0x7f58c4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c4004810 0x7f58c4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc0c6310 con 0x7f58c40039f0 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f58c400bb00 con 0x7f58c4004810 2024-04-25T11:20:16.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f58c400bd00 con 0x7f58dc160370 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c40039f0 0x7f58c4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58dbfff640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 -- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 msgr2=0x7f58c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58daffd640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c4004810 0x7f58c4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 -- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c4004810 msgr2=0x7f58c4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c4004810 0x7f58c4004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 -- 172.21.15.136:0/3554358993 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c4003530 con 0x7f58c40039f0 2024-04-25T11:20:16.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58dbfff640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc160370 0x7f58c4005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:16.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58db7fe640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c40039f0 0x7f58c4003e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58d4002240 tx=0x7f58d401d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6322778c4b24e066 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d400a4a0 con 0x7f58c40039f0 2024-04-25T11:20:16.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d40ad650 con 0x7f58c40039f0 2024-04-25T11:20:16.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58da7fc640 1 -- 172.21.15.136:0/3554358993 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d40ada40 con 0x7f58c40039f0 2024-04-25T11:20:16.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 -- 172.21.15.136:0/3554358993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c40039f0 msgr2=0x7f58c4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 --2- 172.21.15.136:0/3554358993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c40039f0 0x7f58c4003e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f58d4002240 tx=0x7f58d401d360 comp rx=0 tx=0).stop 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58dbfff640 1 -- 172.21.15.136:0/3554358993 reap_dead start 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 -- 172.21.15.136:0/3554358993 shutdown_connections 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 -- 172.21.15.136:0/3554358993 >> 172.21.15.136:0/3554358993 conn(0x7f58dc06f9d0 msgr2=0x7f58dc06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 -- 172.21.15.136:0/3554358993 shutdown_connections 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.565+0000 7f58e2048640 1 -- 172.21.15.136:0/3554358993 wait complete. 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 Processor -- start 2024-04-25T11:20:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- start start 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c40039f0 0x7f58dc14fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 0x7f58dc143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc0c9170 con 0x7f58dc06c670 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc0cabc0 con 0x7f58c40039f0 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f58dc1f65e0 con 0x7f58dc15b6b0 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46134/0 (socket says 172.21.15.136:46134) 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 -- 172.21.15.136:0/4182082505 learned_addr learned my addr 172.21.15.136:0/4182082505 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58db7fe640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c40039f0 0x7f58dc14fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58dbfff640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 0x7f58dc143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 -- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 msgr2=0x7f58dc143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 0x7f58dc143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 -- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c40039f0 msgr2=0x7f58dc14fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c40039f0 0x7f58dc14fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 -- 172.21.15.136:0/4182082505 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc1f6110 con 0x7f58dc06c670 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58dbfff640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 0x7f58dc143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58daffd640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58d000d570 tx=0x7f58d000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0005480 con 0x7f58dc06c670 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d0017070 con 0x7f58dc06c670 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0003820 con 0x7f58dc06c670 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58dc06dbc0 con 0x7f58dc06c670 2024-04-25T11:20:16.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.569+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58dc06e180 con 0x7f58dc06c670 2024-04-25T11:20:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58dc06a350 con 0x7f58dc06c670 2024-04-25T11:20:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f58d00039c0 con 0x7f58dc06c670 2024-04-25T11:20:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58c3fff640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 0x7f58c8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f58d008ed40 con 0x7f58dc06c670 2024-04-25T11:20:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58db7fe640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 0x7f58c8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.573+0000 7f58db7fe640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 0x7f58c8071940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58d40ad000 tx=0x7f58d40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.577+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f58d005c000 con 0x7f58dc06c670 2024-04-25T11:20:16.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.729+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f58dc0cb0b0 con 0x7f58dc06c670 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.729+0000 7f58c3fff640 1 -- 172.21.15.136:0/4182082505 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f58d005b900 con 0x7f58dc06c670 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout:[ 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout: "poolnum": 1, 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout: "poolname": ".mgr" 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-25T11:20:16.727 INFO:teuthology.orchestra.run.smithi136.stdout:] 2024-04-25T11:20:16.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 msgr2=0x7f58c8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 0x7f58c8071940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58d40ad000 tx=0x7f58d40a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 msgr2=0x7f58dc1f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58d000d570 tx=0x7f58d000da30 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 shutdown_connections 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f58c806f430 0x7f58c8071940 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f58dc15b6b0 0x7f58dc143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc06c670 0x7f58dc1f5bd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 --2- 172.21.15.136:0/4182082505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f58c40039f0 0x7f58dc14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 >> 172.21.15.136:0/4182082505 conn(0x7f58dc06f9d0 msgr2=0x7f58dc148f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 shutdown_connections 2024-04-25T11:20:16.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.733+0000 7f58e2048640 1 -- 172.21.15.136:0/4182082505 wait complete. 2024-04-25T11:20:16.747 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T11:20:16.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T11:20:16.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.889+0000 7f327701c640 1 Processor -- start 2024-04-25T11:20:16.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.889+0000 7f327701c640 1 -- start start 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f327701c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 0x7f32700c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f327701c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f32700c5c30 0x7f3270163640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f327701c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32700c9ed0 con 0x7f32700c77a0 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f327701c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f32700caa10 con 0x7f32700c5c30 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f327701c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f32700ca170 con 0x7f32700c9ac0 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f32700c5c30 0x7f3270163640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f3274db8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f32700c77a0 0x7f3270159660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41662/0 (socket says 172.21.15.136:41662) 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f3274db8640 1 -- 172.21.15.136:0/1720743616 learned_addr learned my addr 172.21.15.136:0/1720743616 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f326ffff640 1 --2- 172.21.15.136:0/1720743616 >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 0x7f32700c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 -- 172.21.15.136:0/1720743616 >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 msgr2=0x7f32700c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:16.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f326f7fe640 1 -- 172.21.15.136:0/1720743616 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102450688 0 0) 0x7f32700c9ed0 con 0x7f32700c77a0 2024-04-25T11:20:16.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 --2- 172.21.15.136:0/1720743616 >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 0x7f32700c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 -- 172.21.15.136:0/1720743616 >> v1:172.21.15.136:6789/0 conn(0x7f32700c77a0 legacy=0x7f3270159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 -- 172.21.15.136:0/1720743616 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32700ca450 con 0x7f32700c5c30 2024-04-25T11:20:16.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f326ffff640 1 --2- 172.21.15.136:0/1720743616 >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 0x7f32700c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:16.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.893+0000 7f32755b9640 1 --2- 172.21.15.136:0/1720743616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f3270163640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3270071fa0 tx=0x7f326801f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9983dc06dd7879 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f326f7fe640 1 -- 172.21.15.136:0/1720743616 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32680b4070 con 0x7f32700c5c30 2024-04-25T11:20:16.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f326f7fe640 1 -- 172.21.15.136:0/1720743616 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32680af600 con 0x7f32700c5c30 2024-04-25T11:20:16.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f326f7fe640 1 -- 172.21.15.136:0/1720743616 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32680af9d0 con 0x7f32700c5c30 2024-04-25T11:20:16.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- 172.21.15.136:0/1720743616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 msgr2=0x7f3270163640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 --2- 172.21.15.136:0/1720743616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f3270163640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3270071fa0 tx=0x7f326801f110 comp rx=0 tx=0).stop 2024-04-25T11:20:16.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- 172.21.15.136:0/1720743616 shutdown_connections 2024-04-25T11:20:16.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 --2- 172.21.15.136:0/1720743616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f3270163640 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 --2- 172.21.15.136:0/1720743616 >> v2:172.21.15.157:3300/0 conn(0x7f32700c9ac0 0x7f32700c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- 172.21.15.136:0/1720743616 >> 172.21.15.136:0/1720743616 conn(0x7f327006de10 msgr2=0x7f32700ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:16.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- 172.21.15.136:0/1720743616 shutdown_connections 2024-04-25T11:20:16.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- 172.21.15.136:0/1720743616 wait complete. 2024-04-25T11:20:16.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 Processor -- start 2024-04-25T11:20:16.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.897+0000 7f327701c640 1 -- start start 2024-04-25T11:20:16.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32700c77a0 0x7f3270154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32700c9ac0 0x7f32701e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3270159140 con 0x7f32700c5c30 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3270158e60 con 0x7f32700c9ac0 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f327015c2b0 con 0x7f32700c77a0 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46152/0 (socket says 172.21.15.136:46152) 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 -- 172.21.15.136:0/3922751071 learned_addr learned my addr 172.21.15.136:0/3922751071 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 -- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32700c77a0 msgr2=0x7f3270154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f32755b9640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32700c9ac0 0x7f32701e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32700c77a0 0x7f3270154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 -- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32700c9ac0 msgr2=0x7f32701e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32700c9ac0 0x7f32701e35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 -- 172.21.15.136:0/3922751071 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32701f48b0 con 0x7f32700c5c30 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f3274db8640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3260007fd0 tx=0x7f326000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260017070 con 0x7f32700c5c30 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3260005d10 con 0x7f32700c5c30 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32701e24c0 con 0x7f32700c5c30 2024-04-25T11:20:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260007310 con 0x7f32700c5c30 2024-04-25T11:20:16.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32701e2a80 con 0x7f32700c5c30 2024-04-25T11:20:16.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32701e2c50 con 0x7f32700c5c30 2024-04-25T11:20:16.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f326001a070 con 0x7f32700c5c30 2024-04-25T11:20:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.901+0000 7f326d7fa640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 0x7f325c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.905+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f326008e2d0 con 0x7f32700c5c30 2024-04-25T11:20:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.905+0000 7f326ffff640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 0x7f325c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.905+0000 7f326ffff640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 0x7f325c071940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3270153df0 tx=0x7f3264008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:16.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:16.909+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f326005b5a0 con 0x7f32700c5c30 2024-04-25T11:20:17.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.069+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f327006a3c0 con 0x7f32700c5c30 2024-04-25T11:20:17.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.969+0000 7f326d7fa640 1 -- 172.21.15.136:0/3922751071 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f326005aea0 con 0x7f32700c5c30 2024-04-25T11:20:17.967 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_metadata' created 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 msgr2=0x7f325c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 0x7f325c071940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3270153df0 tx=0x7f3264008040 comp rx=0 tx=0).stop 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 msgr2=0x7f32701f4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3260007fd0 tx=0x7f326000c8e0 comp rx=0 tx=0).stop 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 shutdown_connections 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32700c9ac0 0x7f32701e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f325c06f430 0x7f325c071940 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32700c77a0 0x7f3270154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 --2- 172.21.15.136:0/3922751071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32700c5c30 0x7f32701f4370 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 >> 172.21.15.136:0/3922751071 conn(0x7f327006de10 msgr2=0x7f3270152230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 shutdown_connections 2024-04-25T11:20:17.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:17.973+0000 7f327701c640 1 -- 172.21.15.136:0/3922751071 wait complete. 2024-04-25T11:20:18.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T11:20:18.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.165+0000 7ff5eabf7640 1 Processor -- start 2024-04-25T11:20:18.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.165+0000 7ff5eabf7640 1 -- start start 2024-04-25T11:20:18.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5eabf7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff5e4152520 0x7ff5e40736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5eabf7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff5e4073be0 0x7ff5e4162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5eabf7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e40c9a40 con 0x7ff5e40ca870 2024-04-25T11:20:18.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5eabf7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e40c7390 con 0x7ff5e4073be0 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5eabf7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e40c9480 con 0x7ff5e4152520 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff5e4073be0 0x7ff5e4162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e8993640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff5e40ca870 0x7ff5e40c83c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41678/0 (socket says 172.21.15.136:41678) 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e8993640 1 -- 172.21.15.136:0/487452755 learned_addr learned my addr 172.21.15.136:0/487452755 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e3fff640 1 --2- 172.21.15.136:0/487452755 >> v2:172.21.15.157:3300/0 conn(0x7ff5e4152520 0x7ff5e40736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 -- 172.21.15.136:0/487452755 >> v2:172.21.15.157:3300/0 conn(0x7ff5e4152520 msgr2=0x7ff5e40736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 --2- 172.21.15.136:0/487452755 >> v2:172.21.15.157:3300/0 conn(0x7ff5e4152520 0x7ff5e40736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 -- 172.21.15.136:0/487452755 >> v1:172.21.15.136:6789/0 conn(0x7ff5e40ca870 legacy=0x7ff5e40c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 -- 172.21.15.136:0/487452755 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e40c9760 con 0x7ff5e4073be0 2024-04-25T11:20:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e9194640 1 --2- 172.21.15.136:0/487452755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4073be0 0x7ff5e4162cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5e406e170 tx=0x7ff5dc01c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68eae9cd898d71c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e37fe640 1 -- 172.21.15.136:0/487452755 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0ac070 con 0x7ff5e4073be0 2024-04-25T11:20:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e37fe640 1 -- 172.21.15.136:0/487452755 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5dc0a66a0 con 0x7ff5e4073be0 2024-04-25T11:20:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.169+0000 7ff5e37fe640 1 -- 172.21.15.136:0/487452755 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0a6a70 con 0x7ff5e4073be0 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- 172.21.15.136:0/487452755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4073be0 msgr2=0x7ff5e4162cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/487452755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4073be0 0x7ff5e4162cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5e406e170 tx=0x7ff5dc01c7c0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- 172.21.15.136:0/487452755 shutdown_connections 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/487452755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4073be0 0x7ff5e4162cb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/487452755 >> v2:172.21.15.157:3300/0 conn(0x7ff5e4152520 0x7ff5e40736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- 172.21.15.136:0/487452755 >> 172.21.15.136:0/487452755 conn(0x7ff5e40c9d50 msgr2=0x7ff5e40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- 172.21.15.136:0/487452755 shutdown_connections 2024-04-25T11:20:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- 172.21.15.136:0/487452755 wait complete. 2024-04-25T11:20:18.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 Processor -- start 2024-04-25T11:20:18.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- start start 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 0x7ff5e413db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 0x7ff5e4150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e40c48d0 con 0x7ff5e4152520 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e40c4750 con 0x7ff5e40ca870 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5eabf7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e4074250 con 0x7ff5e4073be0 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5e9194640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5e9194640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46180/0 (socket says 172.21.15.136:46180) 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.173+0000 7ff5e9194640 1 -- 172.21.15.136:0/897985513 learned_addr learned my addr 172.21.15.136:0/897985513 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e8993640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 0x7ff5e413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e3fff640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 0x7ff5e4150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 -- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 msgr2=0x7ff5e413db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 0x7ff5e413db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 -- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 msgr2=0x7ff5e4150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 0x7ff5e4150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 -- 172.21.15.136:0/897985513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e4074c00 con 0x7ff5e4152520 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e3fff640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 0x7ff5e4150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e8993640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 0x7ff5e413db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e9194640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc01ae70 tx=0x7ff5dc019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0ac070 con 0x7ff5e4152520 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5dc0a7690 con 0x7ff5e4152520 2024-04-25T11:20:18.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e41f1e90 con 0x7ff5e4152520 2024-04-25T11:20:18.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0a7a60 con 0x7ff5e4152520 2024-04-25T11:20:18.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e413f100 con 0x7ff5e4152520 2024-04-25T11:20:18.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e406a350 con 0x7ff5e4152520 2024-04-25T11:20:18.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff5dc0af070 con 0x7ff5e4152520 2024-04-25T11:20:18.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 0x7ff5c8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:18.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7ff5dc129180 con 0x7ff5e4152520 2024-04-25T11:20:18.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.177+0000 7ff5e8993640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 0x7ff5c8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:18.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.181+0000 7ff5e8993640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 0x7ff5c8071910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8000f70 tx=0x7ff5d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:18.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.185+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff5dc0c0060 con 0x7ff5e4152520 2024-04-25T11:20:18.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.349+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff5e406ee70 con 0x7ff5e4152520 2024-04-25T11:20:18.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.981+0000 7ff5e17fa640 1 -- 172.21.15.136:0/897985513 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7ff5dc016440 con 0x7ff5e4152520 2024-04-25T11:20:18.979 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_data' created 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.985+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 msgr2=0x7ff5c8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.985+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 0x7ff5c8071910 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8000f70 tx=0x7ff5d8009040 comp rx=0 tx=0).stop 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.985+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 msgr2=0x7ff5e4151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.985+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc01ae70 tx=0x7ff5dc019560 comp rx=0 tx=0).stop 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 shutdown_connections 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e4152520 0x7ff5e4151690 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e40ca870 0x7ff5e4150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5c806f400 0x7ff5c8071910 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 --2- 172.21.15.136:0/897985513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5e4073be0 0x7ff5e413db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:18.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 >> 172.21.15.136:0/897985513 conn(0x7ff5e40c9d50 msgr2=0x7ff5e406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:18.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 shutdown_connections 2024-04-25T11:20:18.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:18.989+0000 7ff5eabf7640 1 -- 172.21.15.136:0/897985513 wait complete. 2024-04-25T11:20:19.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T11:20:19.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 Processor -- start 2024-04-25T11:20:19.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 -- start start 2024-04-25T11:20:19.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 0x7fe4d4142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe4d4072dd0 0x7fe4d40c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4d415d590 con 0x7fe4d40c8190 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe4d415d340 con 0x7fe4d4072dd0 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4dc485640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d415e1a0 con 0x7fe4d40729c0 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe4d4072dd0 0x7fe4d40c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4daa22640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe4d40c8190 0x7fe4d4073590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41694/0 (socket says 172.21.15.136:41694) 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe4d4072dd0 0x7fe4d40c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46190/0 (socket says 172.21.15.136:46190) 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4daa22640 1 -- 172.21.15.136:0/2894926546 learned_addr learned my addr 172.21.15.136:0/2894926546 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4da221640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 0x7fe4d4142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 -- 172.21.15.136:0/2894926546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 msgr2=0x7fe4d4142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 0x7fe4d4142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 -- 172.21.15.136:0/2894926546 >> v1:172.21.15.136:6789/0 conn(0x7fe4d40c8190 legacy=0x7fe4d4073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d9a20640 1 -- 172.21.15.136:0/2894926546 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d415d870 con 0x7fe4d4072dd0 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4d921f640 1 -- 172.21.15.136:0/2894926546 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203050118 0 0) 0x7fe4d415d590 con 0x7fe4d40c8190 2024-04-25T11:20:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.173+0000 7fe4da221640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 0x7fe4d4142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:19.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:20:19.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4d9a20640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d40c7c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40097c0 tx=0x7fe4c401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1350263398b7bad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:19.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4d921f640 1 -- 172.21.15.136:0/2894926546 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40b2070 con 0x7fe4d4072dd0 2024-04-25T11:20:19.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4d921f640 1 -- 172.21.15.136:0/2894926546 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c40ad640 con 0x7fe4d4072dd0 2024-04-25T11:20:19.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4d921f640 1 -- 172.21.15.136:0/2894926546 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40ada10 con 0x7fe4d4072dd0 2024-04-25T11:20:19.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 -- 172.21.15.136:0/2894926546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 msgr2=0x7fe4d40c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:19.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d40c7c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40097c0 tx=0x7fe4c401c4a0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 -- 172.21.15.136:0/2894926546 shutdown_connections 2024-04-25T11:20:19.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d40c7c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 --2- 172.21.15.136:0/2894926546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40729c0 0x7fe4d4142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 -- 172.21.15.136:0/2894926546 >> 172.21.15.136:0/2894926546 conn(0x7fe4d40c9d50 msgr2=0x7fe4d40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:19.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 -- 172.21.15.136:0/2894926546 shutdown_connections 2024-04-25T11:20:19.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.177+0000 7fe4dc485640 1 -- 172.21.15.136:0/2894926546 wait complete. 2024-04-25T11:20:19.179 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 Processor -- start 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- start start 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4d40729c0 0x7fe4d4156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40c8190 0x7fe4d413b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d4143370 con 0x7fe4d4072dd0 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41431f0 con 0x7fe4d40729c0 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d4166ef0 con 0x7fe4d40c8190 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46202/0 (socket says 172.21.15.136:46202) 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 -- 172.21.15.136:0/546467412 learned_addr learned my addr 172.21.15.136:0/546467412 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4daa22640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40c8190 0x7fe4d413b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4da221640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4d40729c0 0x7fe4d4156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 -- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40c8190 msgr2=0x7fe4d413b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40c8190 0x7fe4d413b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 -- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4d40729c0 msgr2=0x7fe4d4156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4d40729c0 0x7fe4d4156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 -- 172.21.15.136:0/546467412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d415b730 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4d9a20640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40b0040 tx=0x7fe4c4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40bd070 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c4004030 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d415c360 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c40b2040 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d41f7650 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.181+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d406a350 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.185+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe4c40c0070 con 0x7fe4d4072dd0 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.185+0000 7fe4caffd640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 0x7fe4b8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:19.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.185+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fe4c4128500 con 0x7fe4d4072dd0 2024-04-25T11:20:19.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.185+0000 7fe4da221640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 0x7fe4b8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:19.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.185+0000 7fe4da221640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 0x7fe4b8071940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000b930 tx=0x7fe4d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:19.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.189+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe4c4016090 con 0x7fe4d4072dd0 2024-04-25T11:20:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:19.385+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe4d41f73e0 con 0x7fe4d4072dd0 2024-04-25T11:20:20.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.029+0000 7fe4caffd640 1 -- 172.21.15.136:0/546467412 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fe4c40f5450 con 0x7fe4d4072dd0 2024-04-25T11:20:20.026 INFO:teuthology.orchestra.run.smithi136.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 msgr2=0x7fe4b8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 0x7fe4b8071940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000b930 tx=0x7fe4d0008040 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 msgr2=0x7fe4d4156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4c40b0040 tx=0x7fe4c4019240 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 shutdown_connections 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe4b806f430 0x7fe4b8071940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe4d40c8190 0x7fe4d413b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4d4072dd0 0x7fe4d4156a90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 --2- 172.21.15.136:0/546467412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe4d40729c0 0x7fe4d4156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 >> 172.21.15.136:0/546467412 conn(0x7fe4d40c9d50 msgr2=0x7fe4d4139450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 shutdown_connections 2024-04-25T11:20:20.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.037+0000 7fe4dc485640 1 -- 172.21.15.136:0/546467412 wait complete. 2024-04-25T11:20:20.060 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T11:20:20.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T11:20:20.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 Processor -- start 2024-04-25T11:20:20.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 -- start start 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52e00c6510 0x7f52e0156320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 0x7f52e00d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e0160250 con 0x7f52e015c2d0 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f52e00ccff0 con 0x7f52e00c6510 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e8411640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52e00cd2d0 con 0x7f52e00d1930 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52e00c6510 0x7f52e0156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52e00c6510 0x7f52e0156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46216/0 (socket says 172.21.15.136:46216) 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 -- 172.21.15.136:0/4144734907 learned_addr learned my addr 172.21.15.136:0/4144734907 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e59ac640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 0x7f52e00d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 -- 172.21.15.136:0/4144734907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 msgr2=0x7f52e00d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 0x7f52e00d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 -- 172.21.15.136:0/4144734907 >> v1:172.21.15.136:6789/0 conn(0x7f52e015c2d0 legacy=0x7f52e015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e61ad640 1 -- 172.21.15.136:0/4144734907 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e0160530 con 0x7f52e00c6510 2024-04-25T11:20:20.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.217+0000 7f52e59ac640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 0x7f52e00d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:20.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e61ad640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e00c6510 0x7f52e0156320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009870 tx=0x7f52dc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e20c3a059960b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:20.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e51ab640 1 -- 172.21.15.136:0/4144734907 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0a7070 con 0x7f52e00c6510 2024-04-25T11:20:20.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e51ab640 1 -- 172.21.15.136:0/4144734907 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc0b0070 con 0x7f52e00c6510 2024-04-25T11:20:20.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e51ab640 1 -- 172.21.15.136:0/4144734907 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0ab890 con 0x7f52e00c6510 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 -- 172.21.15.136:0/4144734907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e00c6510 msgr2=0x7f52e0156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e00c6510 0x7f52e0156320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009870 tx=0x7f52dc01c560 comp rx=0 tx=0).stop 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 -- 172.21.15.136:0/4144734907 shutdown_connections 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00d1930 0x7f52e00d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 --2- 172.21.15.136:0/4144734907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e00c6510 0x7f52e0156320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 -- 172.21.15.136:0/4144734907 >> 172.21.15.136:0/4144734907 conn(0x7f52e006e110 msgr2=0x7f52e006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 -- 172.21.15.136:0/4144734907 shutdown_connections 2024-04-25T11:20:20.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.221+0000 7f52e8411640 1 -- 172.21.15.136:0/4144734907 wait complete. 2024-04-25T11:20:20.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 Processor -- start 2024-04-25T11:20:20.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- start start 2024-04-25T11:20:20.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00c6510 0x7f52e00cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e00d1930 0x7f52e014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f52e01695e0 con 0x7f52e015c2d0 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f52e0169460 con 0x7f52e00d1930 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52e0169760 con 0x7f52e00c6510 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46224/0 (socket says 172.21.15.136:46224) 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 -- 172.21.15.136:0/3897612535 learned_addr learned my addr 172.21.15.136:0/3897612535 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e61ad640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00c6510 0x7f52e00cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e59ac640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e00d1930 0x7f52e014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 -- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00c6510 msgr2=0x7f52e00cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00c6510 0x7f52e00cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 -- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e00d1930 msgr2=0x7f52e014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e00d1930 0x7f52e014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 -- 172.21.15.136:0/3897612535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e00cfe30 con 0x7f52e015c2d0 2024-04-25T11:20:20.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e69ae640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52d800c760 tx=0x7f52d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:20.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8003b80 con 0x7f52e015c2d0 2024-04-25T11:20:20.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d8003d20 con 0x7f52e015c2d0 2024-04-25T11:20:20.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e00d0760 con 0x7f52e015c2d0 2024-04-25T11:20:20.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d800aa90 con 0x7f52e015c2d0 2024-04-25T11:20:20.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e00d0d00 con 0x7f52e015c2d0 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.225+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e006a3c0 con 0x7f52e015c2d0 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.229+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f52d800ac30 con 0x7f52e015c2d0 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.229+0000 7f52d6ffd640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 0x7f52c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.229+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f52d808f1e0 con 0x7f52e015c2d0 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.229+0000 7f52e61ad640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 0x7f52c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.229+0000 7f52e61ad640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 0x7f52c4071910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0af000 tx=0x7f52dc0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:20.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.233+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f52d805c0d0 con 0x7f52e015c2d0 2024-04-25T11:20:20.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.401+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f52e00cee80 con 0x7f52e015c2d0 2024-04-25T11:20:20.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.985+0000 7f52d6ffd640 1 -- 172.21.15.136:0/3897612535 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f52d8064150 con 0x7f52e015c2d0 2024-04-25T11:20:20.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 msgr2=0x7f52c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 0x7f52c4071910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0af000 tx=0x7f52dc0a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:20.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 msgr2=0x7f52e01f8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:20.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52d800c760 tx=0x7f52d800cc20 comp rx=0 tx=0).stop 2024-04-25T11:20:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 shutdown_connections 2024-04-25T11:20:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52e015c2d0 0x7f52e01f8af0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52e00d1930 0x7f52e014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f52c406f400 0x7f52c4071910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 --2- 172.21.15.136:0/3897612535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52e00c6510 0x7f52e00cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 >> 172.21.15.136:0/3897612535 conn(0x7f52e006e110 msgr2=0x7f52e00cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 shutdown_connections 2024-04-25T11:20:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:20.993+0000 7f52e8411640 1 -- 172.21.15.136:0/3897612535 wait complete. 2024-04-25T11:20:21.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T11:20:21.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.177+0000 7f5a41b52640 1 Processor -- start 2024-04-25T11:20:21.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.177+0000 7f5a41b52640 1 -- start start 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a41b52640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a41b52640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a41b52640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3c0c63e0 con 0x7f5a3c06fea0 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a41b52640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5a3c0c4b20 con 0x7f5a3c15b2a0 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a41b52640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c0c4e00 con 0x7f5a3c070cc0 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46238/0 (socket says 172.21.15.136:46238) 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 -- 172.21.15.136:0/2399966837 learned_addr learned my addr 172.21.15.136:0/2399966837 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3affd640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 -- 172.21.15.136:0/2399966837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 msgr2=0x7f5a3c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 -- 172.21.15.136:0/2399966837 >> v1:172.21.15.157:6789/0 conn(0x7f5a3c06fea0 legacy=0x7f5a3c15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:21.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 -- 172.21.15.136:0/2399966837 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c0c50e0 con 0x7f5a3c15b2a0 2024-04-25T11:20:21.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3b7fe640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009870 tx=0x7f5a2c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd6e9600f8d9fb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:21.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3a7fc640 1 -- 172.21.15.136:0/2399966837 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0ab650 con 0x7f5a3c15b2a0 2024-04-25T11:20:21.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3a7fc640 1 -- 172.21.15.136:0/2399966837 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a2c0b0070 con 0x7f5a3c15b2a0 2024-04-25T11:20:21.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.181+0000 7f5a3a7fc640 1 -- 172.21.15.136:0/2399966837 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0aba10 con 0x7f5a3c15b2a0 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- 172.21.15.136:0/2399966837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c15b2a0 msgr2=0x7f5a3c0c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009870 tx=0x7f5a2c01c560 comp rx=0 tx=0).stop 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- 172.21.15.136:0/2399966837 shutdown_connections 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- 172.21.15.136:0/2399966837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c15b2a0 0x7f5a3c0c5ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- 172.21.15.136:0/2399966837 >> 172.21.15.136:0/2399966837 conn(0x7f5a3c06cd00 msgr2=0x7f5a3c0c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- 172.21.15.136:0/2399966837 shutdown_connections 2024-04-25T11:20:21.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- 172.21.15.136:0/2399966837 wait complete. 2024-04-25T11:20:21.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 Processor -- start 2024-04-25T11:20:21.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- start start 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c06fea0 0x7f5a3c163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c1644b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0c47c0 con 0x7f5a3c165150 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0706c0 con 0x7f5a3c06fea0 2024-04-25T11:20:21.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a41b52640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c154150 con 0x7f5a3c070cc0 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46252/0 (socket says 172.21.15.136:46252) 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.185+0000 7f5a3bfff640 1 -- 172.21.15.136:0/3329255249 learned_addr learned my addr 172.21.15.136:0/3329255249 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3affd640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c1644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3b7fe640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c06fea0 0x7f5a3c163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 -- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 msgr2=0x7f5a3c1644b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c1644b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 -- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c06fea0 msgr2=0x7f5a3c163050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c06fea0 0x7f5a3c163050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 -- 172.21.15.136:0/3329255249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c162d60 con 0x7f5a3c165150 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3bfff640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c760 tx=0x7f5a3400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34003b80 con 0x7f5a3c165150 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a34003d20 con 0x7f5a3c165150 2024-04-25T11:20:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3c163590 con 0x7f5a3c165150 2024-04-25T11:20:21.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400aa90 con 0x7f5a3c165150 2024-04-25T11:20:21.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3c072e80 con 0x7f5a3c165150 2024-04-25T11:20:21.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3c162ab0 con 0x7f5a3c165150 2024-04-25T11:20:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5a3400ac30 con 0x7f5a3c165150 2024-04-25T11:20:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 0x7f5a28071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f5a3408f1e0 con 0x7f5a3c165150 2024-04-25T11:20:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.189+0000 7f5a3b7fe640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 0x7f5a28071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.193+0000 7f5a3b7fe640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 0x7f5a28071940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0af000 tx=0x7f5a2c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:21.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.197+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5a3405c030 con 0x7f5a3c165150 2024-04-25T11:20:21.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:21.357+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5a3c0c44a0 con 0x7f5a3c165150 2024-04-25T11:20:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.041+0000 7f5a23fff640 1 -- 172.21.15.136:0/3329255249 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f5a3405b930 con 0x7f5a3c165150 2024-04-25T11:20:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_data_ec' created 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 msgr2=0x7f5a28071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 0x7f5a28071940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0af000 tx=0x7f5a2c0a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 msgr2=0x7f5a3c151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c760 tx=0x7f5a3400cc20 comp rx=0 tx=0).stop 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 shutdown_connections 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a3c165150 0x7f5a3c151a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5a2806f430 0x7f5a28071940 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0af000 tx=0x7f5a2c0a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5a3c070cc0 0x7f5a3c1644b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 --2- 172.21.15.136:0/3329255249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a3c06fea0 0x7f5a3c163050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 >> 172.21.15.136:0/3329255249 conn(0x7f5a3c06cd00 msgr2=0x7f5a3c0c90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 shutdown_connections 2024-04-25T11:20:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.049+0000 7f5a41b52640 1 -- 172.21.15.136:0/3329255249 wait complete. 2024-04-25T11:20:23.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T11:20:23.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 Processor -- start 2024-04-25T11:20:23.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 -- start start 2024-04-25T11:20:23.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f43301601c0 0x7f43300c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4330155e20 0x7f4330156230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f433015c930 con 0x7f433015c0f0 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f43301605d0 con 0x7f4330155e20 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.237+0000 7f43385c0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f433015fd30 con 0x7f43301601c0 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4330155e20 0x7f4330156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4335b5b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f433015c0f0 0x7f433006c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41710/0 (socket says 172.21.15.136:41710) 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4335b5b640 1 -- 172.21.15.136:0/2260699462 learned_addr learned my addr 172.21.15.136:0/2260699462 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f433635c640 1 --2- 172.21.15.136:0/2260699462 >> v2:172.21.15.157:3300/0 conn(0x7f43301601c0 0x7f43300c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 -- 172.21.15.136:0/2260699462 >> v2:172.21.15.157:3300/0 conn(0x7f43301601c0 msgr2=0x7f43300c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 --2- 172.21.15.136:0/2260699462 >> v2:172.21.15.157:3300/0 conn(0x7f43301601c0 0x7f43300c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 -- 172.21.15.136:0/2260699462 >> v1:172.21.15.136:6789/0 conn(0x7f433015c0f0 legacy=0x7f433006c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 -- 172.21.15.136:0/2260699462 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433015cc10 con 0x7f4330155e20 2024-04-25T11:20:23.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f433535a640 1 -- 172.21.15.136:0/2260699462 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3152862673 0 0) 0x7f433015c930 con 0x7f433015c0f0 2024-04-25T11:20:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f4336b5d640 1 --2- 172.21.15.136:0/2260699462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4330155e20 0x7f4330156230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43300cbe20 tx=0x7f432c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e34cacb1f2cf86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f433535a640 1 -- 172.21.15.136:0/2260699462 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0b5070 con 0x7f4330155e20 2024-04-25T11:20:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f433535a640 1 -- 172.21.15.136:0/2260699462 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432c0a8970 con 0x7f4330155e20 2024-04-25T11:20:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f433535a640 1 -- 172.21.15.136:0/2260699462 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0a8d40 con 0x7f4330155e20 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 -- 172.21.15.136:0/2260699462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4330155e20 msgr2=0x7f4330156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 --2- 172.21.15.136:0/2260699462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4330155e20 0x7f4330156230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43300cbe20 tx=0x7f432c01ec70 comp rx=0 tx=0).stop 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 -- 172.21.15.136:0/2260699462 shutdown_connections 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 --2- 172.21.15.136:0/2260699462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4330155e20 0x7f4330156230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 --2- 172.21.15.136:0/2260699462 >> v2:172.21.15.157:3300/0 conn(0x7f43301601c0 0x7f43300c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 -- 172.21.15.136:0/2260699462 >> 172.21.15.136:0/2260699462 conn(0x7f43300ce260 msgr2=0x7f43300ce670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 -- 172.21.15.136:0/2260699462 shutdown_connections 2024-04-25T11:20:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.241+0000 7f43385c0640 1 -- 172.21.15.136:0/2260699462 wait complete. 2024-04-25T11:20:23.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 Processor -- start 2024-04-25T11:20:23.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 -- start start 2024-04-25T11:20:23.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4330155e20 0x7f43301551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f433015c0f0 0x7f43301f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f433015f9c0 con 0x7f43301601c0 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f433015f6e0 con 0x7f4330155e20 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f43385c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4330156770 con 0x7f433015c0f0 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46258/0 (socket says 172.21.15.136:46258) 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 -- 172.21.15.136:0/1701180207 learned_addr learned my addr 172.21.15.136:0/1701180207 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4335b5b640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f433015c0f0 0x7f43301f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f433635c640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4330155e20 0x7f43301551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 -- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f433015c0f0 msgr2=0x7f43301f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f433015c0f0 0x7f43301f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 -- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4330155e20 msgr2=0x7f43301551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4330155e20 0x7f43301551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 -- 172.21.15.136:0/1701180207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43301f63a0 con 0x7f43301601c0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4336b5d640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f432c0b3040 tx=0x7f432c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0bf070 con 0x7f43301601c0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.245+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432c0b0780 con 0x7f43301601c0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43301f55e0 con 0x7f43301601c0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0b5040 con 0x7f43301601c0 2024-04-25T11:20:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43301f5ba0 con 0x7f43301601c0 2024-04-25T11:20:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433006a3c0 con 0x7f43301601c0 2024-04-25T11:20:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f432c0c2070 con 0x7f43301601c0 2024-04-25T11:20:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f4326ffd640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 0x7f4314071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f432c12bbb0 con 0x7f43301601c0 2024-04-25T11:20:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f433635c640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 0x7f4314071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:23.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.249+0000 7f433635c640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 0x7f4314071940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f432000b930 tx=0x7f4320008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:23.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.253+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f432c0187d0 con 0x7f43301601c0 2024-04-25T11:20:23.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:23.413+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f4330153a00 con 0x7f43301601c0 2024-04-25T11:20:24.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.045+0000 7f4326ffd640 1 -- 172.21.15.136:0/1701180207 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f432c0f81d0 con 0x7f43301601c0 2024-04-25T11:20:24.043 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T11:20:24.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 msgr2=0x7f4314071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:24.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 0x7f4314071940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f432000b930 tx=0x7f4320008040 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 msgr2=0x7f43300cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f432c0b3040 tx=0x7f432c01ba10 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 shutdown_connections 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43301601c0 0x7f43300cb1b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f431406f430 0x7f4314071940 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f432000b930 tx=0x7f4320008040 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f433015c0f0 0x7f43301f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 --2- 172.21.15.136:0/1701180207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4330155e20 0x7f43301551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.053+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 >> 172.21.15.136:0/1701180207 conn(0x7f43300ce260 msgr2=0x7f4330152b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.057+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 shutdown_connections 2024-04-25T11:20:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.057+0000 7f43385c0640 1 -- 172.21.15.136:0/1701180207 wait complete. 2024-04-25T11:20:24.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T11:20:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 Processor -- start 2024-04-25T11:20:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 -- start start 2024-04-25T11:20:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6480c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff64815a3d0 0x7ff64815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6480c9af0 con 0x7ff6480c96e0 2024-04-25T11:20:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff64806cc10 con 0x7ff64815a3d0 2024-04-25T11:20:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64e9af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6480cb220 con 0x7ff6480c79e0 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff64815a3d0 0x7ff64815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff64815a3d0 0x7ff64815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46266/0 (socket says 172.21.15.136:46266) 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2896099124 learned_addr learned my addr 172.21.15.136:0/2896099124 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff6477fe640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6480c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2896099124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 msgr2=0x7ff6480c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6480c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2896099124 >> v1:172.21.15.157:6789/0 conn(0x7ff6480c96e0 legacy=0x7ff6481540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.249+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2896099124 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6480cb500 con 0x7ff64815a3d0 2024-04-25T11:20:24.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64cf4c640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff64815a7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f9adb42a12076b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:24.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff646ffd640 1 -- 172.21.15.136:0/2896099124 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400a5670 con 0x7ff64815a3d0 2024-04-25T11:20:24.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff646ffd640 1 -- 172.21.15.136:0/2896099124 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6400a5810 con 0x7ff64815a3d0 2024-04-25T11:20:24.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff646ffd640 1 -- 172.21.15.136:0/2896099124 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400a5be0 con 0x7ff64815a3d0 2024-04-25T11:20:24.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- 172.21.15.136:0/2896099124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 msgr2=0x7ff64815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff64815a7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff64001c9f0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- 172.21.15.136:0/2896099124 shutdown_connections 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff64815a7e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2896099124 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6480c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- 172.21.15.136:0/2896099124 >> 172.21.15.136:0/2896099124 conn(0x7ff64806f9d0 msgr2=0x7ff64806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- 172.21.15.136:0/2896099124 shutdown_connections 2024-04-25T11:20:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- 172.21.15.136:0/2896099124 wait complete. 2024-04-25T11:20:24.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 Processor -- start 2024-04-25T11:20:24.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.253+0000 7ff64e9af640 1 -- start start 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6481450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 0x7ff648146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff6480ced40 con 0x7ff64815a3d0 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff64806d440 con 0x7ff6480c96e0 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff648150400 con 0x7ff6480c79e0 2024-04-25T11:20:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46274/0 (socket says 172.21.15.136:46274) 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2978806659 learned_addr learned my addr 172.21.15.136:0/2978806659 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff6477fe640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 0x7ff648146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff647fff640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6481450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 msgr2=0x7ff6481450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6481450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 msgr2=0x7ff648146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 0x7ff648146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 -- 172.21.15.136:0/2978806659 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff648146560 con 0x7ff64815a3d0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff6477fe640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 0x7ff648146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64cf4c640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff640019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640009510 con 0x7ff64815a3d0 2024-04-25T11:20:24.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6400be070 con 0x7ff64815a3d0 2024-04-25T11:20:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6481f37c0 con 0x7ff64815a3d0 2024-04-25T11:20:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6400b8870 con 0x7ff64815a3d0 2024-04-25T11:20:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6481f3d80 con 0x7ff64815a3d0 2024-04-25T11:20:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64806a350 con 0x7ff64815a3d0 2024-04-25T11:20:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff6400b8a10 con 0x7ff64815a3d0 2024-04-25T11:20:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.257+0000 7ff644ff9640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 0x7ff634071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.261+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7ff64012a670 con 0x7ff64815a3d0 2024-04-25T11:20:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.261+0000 7ff647fff640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 0x7ff634071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.261+0000 7ff647fff640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 0x7ff634071910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff63800a750 tx=0x7ff638008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:24.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.265+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff6400165e0 con 0x7ff64815a3d0 2024-04-25T11:20:24.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:24.449+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7ff648151770 con 0x7ff64815a3d0 2024-04-25T11:20:25.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.069+0000 7ff644ff9640 1 -- 172.21.15.136:0/2978806659 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v5) ==== 137+0+0 (secure 0 0 0) 0x7ff6400f7240 con 0x7ff64815a3d0 2024-04-25T11:20:25.068 INFO:teuthology.orchestra.run.smithi136.stderr:added data pool 4 to fsmap 2024-04-25T11:20:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 msgr2=0x7ff634071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 0x7ff634071910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff63800a750 tx=0x7ff638008040 comp rx=0 tx=0).stop 2024-04-25T11:20:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 msgr2=0x7ff648159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff640009e50 tx=0x7ff640019790 comp rx=0 tx=0).stop 2024-04-25T11:20:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 shutdown_connections 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff64815a3d0 0x7ff648159330 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6480c96e0 0x7ff648146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff63406f400 0x7ff634071910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 --2- 172.21.15.136:0/2978806659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6480c79e0 0x7ff6481450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 >> 172.21.15.136:0/2978806659 conn(0x7ff64806f9d0 msgr2=0x7ff648152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 shutdown_connections 2024-04-25T11:20:25.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.073+0000 7ff64e9af640 1 -- 172.21.15.136:0/2978806659 wait complete. 2024-04-25T11:20:25.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:25.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 Processor -- start 2024-04-25T11:20:25.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 -- start start 2024-04-25T11:20:25.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8800c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff88015aa00 0x7ff8800c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8800c3880 con 0x7ff880154060 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff8800c8fd0 con 0x7ff88015aa00 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff886366640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff88006d9b0 con 0x7ff88015a5f0 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff880154060 0x7ff8800c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53302/0 (socket says 172.21.15.136:53302) 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87ffff640 1 -- 172.21.15.136:0/1972086573 learned_addr learned my addr 172.21.15.136:0/1972086573 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff884903640 1 --2- 172.21.15.136:0/1972086573 >> v2:172.21.15.157:3300/0 conn(0x7ff88015aa00 0x7ff8800c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87f7fe640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8800c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779886636 0 0) 0x7ff8800c3880 con 0x7ff880154060 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff868003530 con 0x7ff880154060 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4132715924 0 0) 0x7ff868003530 con 0x7ff880154060 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.241+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 msgr2=0x7ff8800c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8800c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 >> v2:172.21.15.157:3300/0 conn(0x7ff88015aa00 msgr2=0x7ff8800c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 --2- 172.21.15.136:0/1972086573 >> v2:172.21.15.157:3300/0 conn(0x7ff88015aa00 0x7ff8800c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff880154470 con 0x7ff880154060 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7ff8700016a0 con 0x7ff880154060 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff884903640 1 --2- 172.21.15.136:0/1972086573 >> v2:172.21.15.157:3300/0 conn(0x7ff88015aa00 0x7ff8800c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 >> v1:172.21.15.136:6789/0 conn(0x7ff880154060 legacy=0x7ff8800c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8680039c0 0x7ff868003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff868004810 0x7ff868004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff8800c8fd0 con 0x7ff8680039c0 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff88006d9b0 con 0x7ff868004810 2024-04-25T11:20:25.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff86800bbf0 con 0x7ff88015a5f0 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8680039c0 0x7ff868003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87f7fe640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff868004810 0x7ff868004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff884903640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 -- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 msgr2=0x7ff8680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff88015a5f0 0x7ff8680051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 -- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff868004810 msgr2=0x7ff868004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff868004810 0x7ff868004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 -- 172.21.15.136:0/1972086573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff868003530 con 0x7ff8680039c0 2024-04-25T11:20:25.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87ffff640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8680039c0 0x7ff868003e50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff87000c210 tx=0x7ff87001f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5c967b69c66892 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870007d80 con 0x7ff8680039c0 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff870005ae0 con 0x7ff8680039c0 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff87effd640 1 -- 172.21.15.136:0/1972086573 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870007070 con 0x7ff8680039c0 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 -- 172.21.15.136:0/1972086573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8680039c0 msgr2=0x7ff868003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 --2- 172.21.15.136:0/1972086573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8680039c0 0x7ff868003e50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff87000c210 tx=0x7ff87001f3f0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff884903640 1 -- 172.21.15.136:0/1972086573 reap_dead start 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 -- 172.21.15.136:0/1972086573 shutdown_connections 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 -- 172.21.15.136:0/1972086573 >> 172.21.15.136:0/1972086573 conn(0x7ff8800ca5a0 msgr2=0x7ff8800ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 -- 172.21.15.136:0/1972086573 shutdown_connections 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.245+0000 7ff886366640 1 -- 172.21.15.136:0/1972086573 wait complete. 2024-04-25T11:20:25.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 Processor -- start 2024-04-25T11:20:25.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- start start 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff8680039c0 0x7ff880074d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff880150050 0x7ff880147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff88015a0d0 con 0x7ff880150460 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff88006df30 con 0x7ff880150050 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff880159d80 con 0x7ff8680039c0 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55286/0 (socket says 172.21.15.136:55286) 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 -- 172.21.15.136:0/116144691 learned_addr learned my addr 172.21.15.136:0/116144691 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff87ffff640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff8680039c0 0x7ff880074d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff87f7fe640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff880150050 0x7ff880147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 -- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff8680039c0 msgr2=0x7ff880074d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff8680039c0 0x7ff880074d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 -- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff880150050 msgr2=0x7ff880147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff880150050 0x7ff880147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 -- 172.21.15.136:0/116144691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8801574c0 con 0x7ff880150460 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff884903640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8800cb320 tx=0x7ff878004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878019070 con 0x7ff880150460 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff878007c50 con 0x7ff880150460 2024-04-25T11:20:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8801f03f0 con 0x7ff880150460 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878014400 con 0x7ff880150460 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.249+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8801f06b0 con 0x7ff880150460 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88006a350 con 0x7ff880150460 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff87801c070 con 0x7ff880150460 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff87cff9640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 0x7ff86c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff87808efa0 con 0x7ff880150460 2024-04-25T11:20:25.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff87ffff640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 0x7ff86c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.253+0000 7ff87ffff640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 0x7ff86c071940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ae000 tx=0x7ff870001ae0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.257+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff8801574c0 con 0x7ff880150460 2024-04-25T11:20:25.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.437+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8801f4410 con 0x7ff880150460 2024-04-25T11:20:25.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.437+0000 7ff87cff9640 1 -- 172.21.15.136:0/116144691 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+6829 (secure 0 0 0) 0x7ff87805b5a0 con 0x7ff880150460 2024-04-25T11:20:25.436 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:25.436 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:20:24.451885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-25T11:20:25.436 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 5 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 msgr2=0x7ff86c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 0x7ff86c071940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ae000 tx=0x7ff870001ae0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 msgr2=0x7ff8800709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8800cb320 tx=0x7ff878004840 comp rx=0 tx=0).stop 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 shutdown_connections 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff880150460 0x7ff8800709c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff880150050 0x7ff880147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff86c06f430 0x7ff86c071940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 --2- 172.21.15.136:0/116144691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff8680039c0 0x7ff880074d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 >> 172.21.15.136:0/116144691 conn(0x7ff8800ca5a0 msgr2=0x7ff88006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 shutdown_connections 2024-04-25T11:20:25.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.441+0000 7ff886366640 1 -- 172.21.15.136:0/116144691 wait complete. 2024-04-25T11:20:25.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:25.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 Processor -- start 2024-04-25T11:20:25.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 -- start start 2024-04-25T11:20:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0f0070c80 0x7fa0f01657a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f0143580 con 0x7fa0f0070870 2024-04-25T11:20:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f015dd20 con 0x7fa0f0143170 2024-04-25T11:20:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0f508d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f015e000 con 0x7fa0f0070c80 2024-04-25T11:20:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0ef59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0f0070c80 0x7fa0f01657a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0e659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa0f0070870 0x7fa0f00c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53318/0 (socket says 172.21.15.136:53318) 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0e659c640 1 -- 172.21.15.136:0/1524877218 learned_addr learned my addr 172.21.15.136:0/1524877218 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0ef59e640 1 -- 172.21.15.136:0/1524877218 >> v1:172.21.15.157:6789/0 conn(0x7fa0f0143170 legacy=0x7fa0f015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0ef59e640 1 -- 172.21.15.136:0/1524877218 >> v1:172.21.15.136:6789/0 conn(0x7fa0f0070870 legacy=0x7fa0f00c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0ef59e640 1 -- 172.21.15.136:0/1524877218 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f006fd90 con 0x7fa0f0070c80 2024-04-25T11:20:25.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.593+0000 7fa0ee59c640 1 -- 172.21.15.136:0/1524877218 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954187336 0 0) 0x7fa0d8001120 con 0x7fa0f0070870 2024-04-25T11:20:25.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0ef59e640 1 --2- 172.21.15.136:0/1524877218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f01657a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009d90 tx=0x7fa0e801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d333be18888c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0ee59c640 1 -- 172.21.15.136:0/1524877218 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80b4070 con 0x7fa0f0070c80 2024-04-25T11:20:25.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0ee59c640 1 -- 172.21.15.136:0/1524877218 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e80af600 con 0x7fa0f0070c80 2024-04-25T11:20:25.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0ee59c640 1 -- 172.21.15.136:0/1524877218 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80af9d0 con 0x7fa0f0070c80 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 -- 172.21.15.136:0/1524877218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 msgr2=0x7fa0f01657a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 --2- 172.21.15.136:0/1524877218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f01657a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009d90 tx=0x7fa0e801f150 comp rx=0 tx=0).stop 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 -- 172.21.15.136:0/1524877218 shutdown_connections 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 --2- 172.21.15.136:0/1524877218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f01657a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 -- 172.21.15.136:0/1524877218 >> 172.21.15.136:0/1524877218 conn(0x7fa0f00c9d50 msgr2=0x7fa0f00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 -- 172.21.15.136:0/1524877218 shutdown_connections 2024-04-25T11:20:25.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 -- 172.21.15.136:0/1524877218 wait complete. 2024-04-25T11:20:25.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.597+0000 7fa0f508d640 1 Processor -- start 2024-04-25T11:20:25.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- start start 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0f0070870 0x7fa0f0138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0f0143170 0x7fa0f00c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f015e2e0 con 0x7fa0f0070c80 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f0070350 con 0x7fa0f0143170 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f00712a0 con 0x7fa0f0070870 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55310/0 (socket says 172.21.15.136:55310) 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 -- 172.21.15.136:0/3944346161 learned_addr learned my addr 172.21.15.136:0/3944346161 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0ef59e640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0f0143170 0x7fa0f00c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0eed9d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0f0070870 0x7fa0f0138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 -- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0f0070870 msgr2=0x7fa0f0138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0f0070870 0x7fa0f0138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 -- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0f0143170 msgr2=0x7fa0f00c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0f0143170 0x7fa0f00c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 -- 172.21.15.136:0/3944346161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f00cc8a0 con 0x7fa0f0070c80 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e659c640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c410 tx=0x7fa0d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d80074e0 con 0x7fa0f0070c80 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d80047f0 con 0x7fa0f0070c80 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8007680 con 0x7fa0f0070c80 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01420b0 con 0x7fa0f0070c80 2024-04-25T11:20:25.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f01408a0 con 0x7fa0f0070c80 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f006a350 con 0x7fa0f0070c80 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.601+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa0d8011070 con 0x7fa0f0070c80 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.605+0000 7fa0e7fff640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 0x7fa0d0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.605+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa0d808f0d0 con 0x7fa0f0070c80 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.605+0000 7fa0eed9d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 0x7fa0d0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.605+0000 7fa0eed9d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 0x7fa0d0071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0000f70 tx=0x7fa0e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.609+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa0f0143580 con 0x7fa0f0070c80 2024-04-25T11:20:25.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.757+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0f013e590 con 0x7fa0f0070c80 2024-04-25T11:20:25.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0e7fff640 1 -- 172.21.15.136:0/3944346161 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fa0d805b750 con 0x7fa0f0070c80 2024-04-25T11:20:25.757 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:25.757 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 msgr2=0x7fa0d0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 0x7fa0d0071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0000f70 tx=0x7fa0e0009040 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 msgr2=0x7fa0f0141250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c410 tx=0x7fa0d800c8d0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 shutdown_connections 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0f0143170 0x7fa0f00c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f0070c80 0x7fa0f0141250 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa0d006f400 0x7fa0d0071910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 --2- 172.21.15.136:0/3944346161 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa0f0070870 0x7fa0f0138ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 >> 172.21.15.136:0/3944346161 conn(0x7fa0f00c9d50 msgr2=0x7fa0f0153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 shutdown_connections 2024-04-25T11:20:25.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.761+0000 7fa0f508d640 1 -- 172.21.15.136:0/3944346161 wait complete. 2024-04-25T11:20:25.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:25.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.925+0000 7f814b358640 1 Processor -- start 2024-04-25T11:20:25.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 -- start start 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8144153190 0x7f81440c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 0x7f8144158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81441596f0 con 0x7f81441592e0 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f81440c8370 con 0x7f8144153190 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f814b358640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f81440c8090 con 0x7f81440c9570 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8144153190 0x7f81440c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8144153190 0x7f81440c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55316/0 (socket says 172.21.15.136:55316) 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 -- 172.21.15.136:0/1747632418 learned_addr learned my addr 172.21.15.136:0/1747632418 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81488f3640 1 --2- 172.21.15.136:0/1747632418 >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 0x7f8144158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 -- 172.21.15.136:0/1747632418 >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 msgr2=0x7f8144158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 --2- 172.21.15.136:0/1747632418 >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 0x7f8144158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 -- 172.21.15.136:0/1747632418 >> v1:172.21.15.157:6789/0 conn(0x7f81441592e0 legacy=0x7f814415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81490f4640 1 -- 172.21.15.136:0/1747632418 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81440c6aa0 con 0x7f8144153190 2024-04-25T11:20:25.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.929+0000 7f81488f3640 1 --2- 172.21.15.136:0/1747632418 >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 0x7f8144158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:25.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f81490f4640 1 --2- 172.21.15.136:0/1747632418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f81440c9030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81441569d0 tx=0x7f814001f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674095a865422e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f813bfff640 1 -- 172.21.15.136:0/1747632418 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81400b4070 con 0x7f8144153190 2024-04-25T11:20:25.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f813bfff640 1 -- 172.21.15.136:0/1747632418 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81400af400 con 0x7f8144153190 2024-04-25T11:20:25.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f813bfff640 1 -- 172.21.15.136:0/1747632418 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81400af7d0 con 0x7f8144153190 2024-04-25T11:20:25.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- 172.21.15.136:0/1747632418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 msgr2=0x7f81440c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 --2- 172.21.15.136:0/1747632418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f81440c9030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81441569d0 tx=0x7f814001f1a0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- 172.21.15.136:0/1747632418 shutdown_connections 2024-04-25T11:20:25.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 --2- 172.21.15.136:0/1747632418 >> v2:172.21.15.157:3300/0 conn(0x7f81440c9570 0x7f8144158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 --2- 172.21.15.136:0/1747632418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f81440c9030 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- 172.21.15.136:0/1747632418 >> 172.21.15.136:0/1747632418 conn(0x7f81440c9d50 msgr2=0x7f81440ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:25.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- 172.21.15.136:0/1747632418 shutdown_connections 2024-04-25T11:20:25.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- 172.21.15.136:0/1747632418 wait complete. 2024-04-25T11:20:25.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 Processor -- start 2024-04-25T11:20:25.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.933+0000 7f814b358640 1 -- start start 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81440c9570 0x7f81441f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81441592e0 0x7f8144138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f81440c7340 con 0x7f8144153190 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81440c7060 con 0x7f81441592e0 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81440ca870 con 0x7f81440c9570 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55320/0 (socket says 172.21.15.136:55320) 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 -- 172.21.15.136:0/2178748907 learned_addr learned my addr 172.21.15.136:0/2178748907 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81490f4640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81440c9570 0x7f81441f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81498f5640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81441592e0 0x7f8144138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 -- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81440c9570 msgr2=0x7f81441f09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81440c9570 0x7f81441f09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 -- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81441592e0 msgr2=0x7f8144138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81441592e0 0x7f8144138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 -- 172.21.15.136:0/2178748907 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81441f0f30 con 0x7f8144153190 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f81488f3640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f813400d830 tx=0x7f813400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134018020 con 0x7f8144153190 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81340096c0 con 0x7f8144153190 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81441f3f70 con 0x7f8144153190 2024-04-25T11:20:25.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134009ab0 con 0x7f8144153190 2024-04-25T11:20:25.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8144145b20 con 0x7f8144153190 2024-04-25T11:20:25.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f814406a350 con 0x7f8144153190 2024-04-25T11:20:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f813401b020 con 0x7f8144153190 2024-04-25T11:20:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 0x7f8128071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.937+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f813408fd20 con 0x7f8144153190 2024-04-25T11:20:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.941+0000 7f81490f4640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 0x7f8128071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:25.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.941+0000 7f81490f4640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 0x7f8128071940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81400b1000 tx=0x7f81400aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:25.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:25.945+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8134022020 con 0x7f8144153190 2024-04-25T11:20:26.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.105+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8144159910 con 0x7f8144153190 2024-04-25T11:20:26.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f8139ffb640 1 -- 172.21.15.136:0/2178748907 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f813405c960 con 0x7f8144153190 2024-04-25T11:20:26.105 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:26.105 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 msgr2=0x7f8128071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 0x7f8128071940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81400b1000 tx=0x7f81400aa000 comp rx=0 tx=0).stop 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 msgr2=0x7f814413ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f813400d830 tx=0x7f813400dcf0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 shutdown_connections 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81441592e0 0x7f8144138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8144153190 0x7f814413ef90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f812806f430 0x7f8128071940 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 --2- 172.21.15.136:0/2178748907 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81440c9570 0x7f81441f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 >> 172.21.15.136:0/2178748907 conn(0x7f81440c9d50 msgr2=0x7f8144146470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:26.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 shutdown_connections 2024-04-25T11:20:26.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.109+0000 7f814b358640 1 -- 172.21.15.136:0/2178748907 wait complete. 2024-04-25T11:20:26.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:26.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 Processor -- start 2024-04-25T11:20:26.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 -- start start 2024-04-25T11:20:26.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9b80156380 0x7f9b8015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b800c6470 con 0x7f9b800c66e0 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b80156790 con 0x7f9b800cdee0 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b887a3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9b800cd390 con 0x7f9b80156380 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b8653f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b800c66e0 0x7f9b8015f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53322/0 (socket says 172.21.15.136:53322) 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b8653f640 1 -- 172.21.15.136:0/2040656438 learned_addr learned my addr 172.21.15.136:0/2040656438 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b85d3e640 1 --2- 172.21.15.136:0/2040656438 >> v2:172.21.15.136:3300/0 conn(0x7f9b80156380 0x7f9b8015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b85d3e640 1 -- 172.21.15.136:0/2040656438 >> v1:172.21.15.157:6789/0 conn(0x7f9b800cdee0 legacy=0x7f9b80166a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b85d3e640 1 -- 172.21.15.136:0/2040656438 >> v1:172.21.15.136:6789/0 conn(0x7f9b800c66e0 legacy=0x7f9b8015f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b8553d640 1 -- 172.21.15.136:0/2040656438 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189465054 0 0) 0x7f9b800c6470 con 0x7f9b800c66e0 2024-04-25T11:20:26.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.285+0000 7f9b85d3e640 1 -- 172.21.15.136:0/2040656438 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8015c0b0 con 0x7f9b80156380 2024-04-25T11:20:26.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b85d3e640 1 --2- 172.21.15.136:0/2040656438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b80156380 0x7f9b8015cb70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097c0 tx=0x7f9b7001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116ff55e54ec9c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b8553d640 1 -- 172.21.15.136:0/2040656438 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b700b0070 con 0x7f9b80156380 2024-04-25T11:20:26.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b8553d640 1 -- 172.21.15.136:0/2040656438 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b700ab600 con 0x7f9b80156380 2024-04-25T11:20:26.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b8553d640 1 -- 172.21.15.136:0/2040656438 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b700ab9d0 con 0x7f9b80156380 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 -- 172.21.15.136:0/2040656438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b80156380 msgr2=0x7f9b8015cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 --2- 172.21.15.136:0/2040656438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b80156380 0x7f9b8015cb70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097c0 tx=0x7f9b7001c460 comp rx=0 tx=0).stop 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 -- 172.21.15.136:0/2040656438 shutdown_connections 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 --2- 172.21.15.136:0/2040656438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b80156380 0x7f9b8015cb70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 -- 172.21.15.136:0/2040656438 >> 172.21.15.136:0/2040656438 conn(0x7f9b8006d6c0 msgr2=0x7f9b8006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 -- 172.21.15.136:0/2040656438 shutdown_connections 2024-04-25T11:20:26.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.289+0000 7f9b887a3640 1 -- 172.21.15.136:0/2040656438 wait complete. 2024-04-25T11:20:26.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 Processor -- start 2024-04-25T11:20:26.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- start start 2024-04-25T11:20:26.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b800c66e0 0x7f9b80153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 0x7f9b801f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800ce700 con 0x7f9b800cdee0 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8015c370 con 0x7f9b80156380 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9b8016a970 con 0x7f9b800c66e0 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55346/0 (socket says 172.21.15.136:55346) 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 -- 172.21.15.136:0/3797407416 learned_addr learned my addr 172.21.15.136:0/3797407416 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b8653f640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b800c66e0 0x7f9b80153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b86d40640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 0x7f9b801f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 -- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b800c66e0 msgr2=0x7f9b80153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b800c66e0 0x7f9b80153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 -- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 msgr2=0x7f9b801f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 0x7f9b801f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 -- 172.21.15.136:0/3797407416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8015e320 con 0x7f9b800cdee0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b86d40640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 0x7f9b801f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b85d3e640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097c0 tx=0x7f9b7001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b700bd070 con 0x7f9b800cdee0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b70004030 con 0x7f9b800cdee0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b700b0040 con 0x7f9b800cdee0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801f92d0 con 0x7f9b800cdee0 2024-04-25T11:20:26.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.293+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801f9850 con 0x7f9b800cdee0 2024-04-25T11:20:26.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b8006a350 con 0x7f9b800cdee0 2024-04-25T11:20:26.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9b70003620 con 0x7f9b800cdee0 2024-04-25T11:20:26.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b76ffd640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 0x7f9b64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9b70129200 con 0x7f9b800cdee0 2024-04-25T11:20:26.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b8653f640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 0x7f9b64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.297+0000 7f9b8653f640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 0x7f9b64071940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c007fd0 tx=0x7f9b7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.301+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9b801f92d0 con 0x7f9b800cdee0 2024-04-25T11:20:26.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b8015d7c0 con 0x7f9b800cdee0 2024-04-25T11:20:26.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b76ffd640 1 -- 172.21.15.136:0/3797407416 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+6829 (secure 0 0 0) 0x7f9b70016050 con 0x7f9b800cdee0 2024-04-25T11:20:26.502 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:26.502 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:20:24.451885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 5 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 msgr2=0x7f9b64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 0x7f9b64071940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c007fd0 tx=0x7f9b7c009040 comp rx=0 tx=0).stop 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 msgr2=0x7f9b8015dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9b700097c0 tx=0x7f9b7001aca0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 shutdown_connections 2024-04-25T11:20:26.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b80156380 0x7f9b801f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b800cdee0 0x7f9b8015dbe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9b6406f430 0x7f9b64071940 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 --2- 172.21.15.136:0/3797407416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b800c66e0 0x7f9b80153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.497+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 >> 172.21.15.136:0/3797407416 conn(0x7f9b8006d6c0 msgr2=0x7f9b80154620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.501+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 shutdown_connections 2024-04-25T11:20:26.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.501+0000 7f9b887a3640 1 -- 172.21.15.136:0/3797407416 wait complete. 2024-04-25T11:20:26.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:26.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.665+0000 7fc6fc296640 1 Processor -- start 2024-04-25T11:20:26.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.665+0000 7fc6fc296640 1 -- start start 2024-04-25T11:20:26.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc6f4143170 0x7fc6f415d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc6f4070870 0x7fc6f40c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070c80 0x7fc6f40c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f4143580 con 0x7fc6f4143170 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f415dd20 con 0x7fc6f4070870 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fc296640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f415e000 con 0x7fc6f4070c80 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc6f4143170 0x7fc6f415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc6f4143170 0x7fc6f415d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55356/0 (socket says 172.21.15.136:55356) 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 -- 172.21.15.136:0/2638465025 learned_addr learned my addr 172.21.15.136:0/2638465025 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa833640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070c80 0x7fc6f40c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6f9831640 1 --2- 172.21.15.136:0/2638465025 >> v2:172.21.15.157:3300/0 conn(0x7fc6f4070870 0x7fc6f40c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 -- 172.21.15.136:0/2638465025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070c80 msgr2=0x7fc6f40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070c80 0x7fc6f40c6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 -- 172.21.15.136:0/2638465025 >> v2:172.21.15.157:3300/0 conn(0x7fc6f4070870 msgr2=0x7fc6f40c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2638465025 >> v2:172.21.15.157:3300/0 conn(0x7fc6f4070870 0x7fc6f40c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.669+0000 7fc6fa032640 1 -- 172.21.15.136:0/2638465025 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f406fd90 con 0x7fc6f4143170 2024-04-25T11:20:26.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4143170 0x7fc6f415d400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4157a80 tx=0x7fc6f001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9ddc83017b24b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6f9030640 1 -- 172.21.15.136:0/2638465025 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f00b5070 con 0x7fc6f4143170 2024-04-25T11:20:26.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6f9030640 1 -- 172.21.15.136:0/2638465025 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f00a89d0 con 0x7fc6f4143170 2024-04-25T11:20:26.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6f9030640 1 -- 172.21.15.136:0/2638465025 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f00a8da0 con 0x7fc6f4143170 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- 172.21.15.136:0/2638465025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4143170 msgr2=0x7fc6f415d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4143170 0x7fc6f415d400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4157a80 tx=0x7fc6f001ed10 comp rx=0 tx=0).stop 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- 172.21.15.136:0/2638465025 shutdown_connections 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070c80 0x7fc6f40c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2638465025 >> v2:172.21.15.157:3300/0 conn(0x7fc6f4070870 0x7fc6f40c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2638465025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4143170 0x7fc6f415d400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- 172.21.15.136:0/2638465025 >> 172.21.15.136:0/2638465025 conn(0x7fc6f40c9d50 msgr2=0x7fc6f40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- 172.21.15.136:0/2638465025 shutdown_connections 2024-04-25T11:20:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- 172.21.15.136:0/2638465025 wait complete. 2024-04-25T11:20:26.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 Processor -- start 2024-04-25T11:20:26.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.673+0000 7fc6fc296640 1 -- start start 2024-04-25T11:20:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070870 0x7fc6f41f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6f4143170 0x7fc6f4138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f415e2e0 con 0x7fc6f4070c80 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f4070350 con 0x7fc6f4143170 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f40712a0 con 0x7fc6f4070870 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55362/0 (socket says 172.21.15.136:55362) 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 -- 172.21.15.136:0/2837529736 learned_addr learned my addr 172.21.15.136:0/2837529736 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070870 0x7fc6f41f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fa833640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6f4143170 0x7fc6f4138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 -- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070870 msgr2=0x7fc6f41f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070870 0x7fc6f41f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 -- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6f4143170 msgr2=0x7fc6f4138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6f4143170 0x7fc6f4138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 -- 172.21.15.136:0/2837529736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f41f5ef0 con 0x7fc6f4070c80 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6f9831640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400d7d0 tx=0x7fc6e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4018020 con 0x7fc6f4070c80 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e40096a0 con 0x7fc6f4070c80 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f415a500 con 0x7fc6f4070c80 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4009a90 con 0x7fc6f4070c80 2024-04-25T11:20:26.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f415aac0 con 0x7fc6f4070c80 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f406a350 con 0x7fc6f4070c80 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.677+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc6e401b020 con 0x7fc6f4070c80 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.681+0000 7fc6eaffd640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 0x7fc6d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.681+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc6e408fc20 con 0x7fc6f4070c80 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.681+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 0x7fc6d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.681+0000 7fc6fa032640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 0x7fc6d8071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00b2000 tx=0x7fc6f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:26.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.685+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc6e4022020 con 0x7fc6f4070c80 2024-04-25T11:20:26.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.833+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6f4140370 con 0x7fc6f4070c80 2024-04-25T11:20:26.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6eaffd640 1 -- 172.21.15.136:0/2837529736 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fc6e405c7d0 con 0x7fc6f4070c80 2024-04-25T11:20:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:26.834 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 msgr2=0x7fc6d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 0x7fc6d8071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6f00b2000 tx=0x7fc6f0008040 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 msgr2=0x7fc6f4073170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400d7d0 tx=0x7fc6e400dc90 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 shutdown_connections 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6f4143170 0x7fc6f4138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6f4070c80 0x7fc6f4073170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6d806f400 0x7fc6d8071910 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 --2- 172.21.15.136:0/2837529736 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc6f4070870 0x7fc6f41f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:26.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 >> 172.21.15.136:0/2837529736 conn(0x7fc6f40c9d50 msgr2=0x7fc6f413e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:26.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 shutdown_connections 2024-04-25T11:20:26.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:26.837+0000 7fc6fc296640 1 -- 172.21.15.136:0/2837529736 wait complete. 2024-04-25T11:20:27.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T11:20:19.386747+0000', 'modified': '2024-04-25T11:20:24.451885+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4303}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4303': {'gid': 4303, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.157:6839/1671830331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 1671830331}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 1671830331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T11:20:27.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4303:up:creating 2024-04-25T11:20:28.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:28.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 Processor -- start 2024-04-25T11:20:28.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 -- start start 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c0c4340 0x7fdc6c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc6c0c3210 con 0x7fdc6c15c300 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdc6c15cae0 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc7154a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6c0c9d20 con 0x7fdc6c0c4340 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55378/0 (socket says 172.21.15.136:55378) 2024-04-25T11:20:28.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 -- 172.21.15.136:0/1868166132 learned_addr learned my addr 172.21.15.136:0/1868166132 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:28.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6b7fe640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c0c4340 0x7fdc6c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 -- 172.21.15.136:0/1868166132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c0c4340 msgr2=0x7fdc6c0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:28.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c0c4340 0x7fdc6c0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 -- 172.21.15.136:0/1868166132 >> v1:172.21.15.157:6789/0 conn(0x7fdc6c15c300 legacy=0x7fdc6c0749a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:28.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.621+0000 7fdc6affd640 1 -- 172.21.15.136:0/1868166132 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c0c4c90 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc6affd640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c15cc60 tx=0x7fdc6401da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1be8a6c21ecaeda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:28.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc69ffb640 1 -- 172.21.15.136:0/1868166132 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc640a6660 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc69ffb640 1 -- 172.21.15.136:0/1868166132 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc640a6800 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc69ffb640 1 -- 172.21.15.136:0/1868166132 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc640a6bd0 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 -- 172.21.15.136:0/1868166132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c2e00 msgr2=0x7fdc6c15bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c15cc60 tx=0x7fdc6401da00 comp rx=0 tx=0).stop 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 -- 172.21.15.136:0/1868166132 shutdown_connections 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c0c4340 0x7fdc6c0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 --2- 172.21.15.136:0/1868166132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c15bdc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 -- 172.21.15.136:0/1868166132 >> 172.21.15.136:0/1868166132 conn(0x7fdc6c0ca5a0 msgr2=0x7fdc6c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 -- 172.21.15.136:0/1868166132 shutdown_connections 2024-04-25T11:20:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.625+0000 7fdc7154a640 1 -- 172.21.15.136:0/1868166132 wait complete. 2024-04-25T11:20:28.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 Processor -- start 2024-04-25T11:20:28.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- start start 2024-04-25T11:20:28.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c1ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 0x7fdc6c148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c0750b0 con 0x7fdc6c0c4340 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c0ca000 con 0x7fdc6c0c2e00 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6c073200 con 0x7fdc6c15c300 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55380/0 (socket says 172.21.15.136:55380) 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 -- 172.21.15.136:0/3280615761 learned_addr learned my addr 172.21.15.136:0/3280615761 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6b7fe640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 0x7fdc6c148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6affd640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c1ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 -- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 msgr2=0x7fdc6c148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 0x7fdc6c148420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 -- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c0c2e00 msgr2=0x7fdc6c1ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c1ea6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 -- 172.21.15.136:0/3280615761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c1f7800 con 0x7fdc6c0c4340 2024-04-25T11:20:28.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6b7fe640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 0x7fdc6c148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc6a7fc640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c340 tx=0x7fdc6000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc600052d0 con 0x7fdc6c0c4340 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc60005470 con 0x7fdc6c0c4340 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c1f3d70 con 0x7fdc6c0c4340 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60007260 con 0x7fdc6c0c4340 2024-04-25T11:20:28.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.629+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1f4330 con 0x7fdc6c0c4340 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6c06a350 con 0x7fdc6c0c4340 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc60007400 con 0x7fdc6c0c4340 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc537fe640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 0x7fdc58071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc6008ea00 con 0x7fdc6c0c4340 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc6affd640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 0x7fdc58071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.633+0000 7fdc6affd640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 0x7fdc58071940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc640af000 tx=0x7fdc6400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:28.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.637+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdc6005b640 con 0x7fdc6c0c4340 2024-04-25T11:20:28.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.817+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc6c073ec0 con 0x7fdc6c0c4340 2024-04-25T11:20:28.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc537fe640 1 -- 172.21.15.136:0/3280615761 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+6829 (secure 0 0 0) 0x7fdc6005af40 con 0x7fdc6c0c4340 2024-04-25T11:20:28.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:28.817 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:20:24.451885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-25T11:20:28.817 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 5 2024-04-25T11:20:28.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 msgr2=0x7fdc58071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 0x7fdc58071940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc640af000 tx=0x7fdc6400d040 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 msgr2=0x7fdc6c1f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000c340 tx=0x7fdc6000c800 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 shutdown_connections 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5806f430 0x7fdc58071940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc6c15c300 0x7fdc6c148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc6c0c4340 0x7fdc6c1f72c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 --2- 172.21.15.136:0/3280615761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc6c0c2e00 0x7fdc6c1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 >> 172.21.15.136:0/3280615761 conn(0x7fdc6c0ca5a0 msgr2=0x7fdc6c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 shutdown_connections 2024-04-25T11:20:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:28.821+0000 7fdc7154a640 1 -- 172.21.15.136:0/3280615761 wait complete. 2024-04-25T11:20:29.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T11:20:19.386747+0000', 'modified': '2024-04-25T11:20:24.451885+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4303}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4303': {'gid': 4303, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.157:6839/1671830331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 1671830331}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 1671830331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T11:20:29.441 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4303:up:creating 2024-04-25T11:20:30.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:30.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 Processor -- start 2024-04-25T11:20:30.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 -- start start 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7db0070870 0x7f7db00c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db0143580 con 0x7f7db0143170 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db015dd20 con 0x7f7db0070c80 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db831b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7db015e000 con 0x7f7db0070870 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db60b7640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7db0143170 0x7f7db015d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53336/0 (socket says 172.21.15.136:53336) 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db60b7640 1 -- 172.21.15.136:0/314614098 learned_addr learned my addr 172.21.15.136:0/314614098 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db58b6640 1 --2- 172.21.15.136:0/314614098 >> v2:172.21.15.136:3300/0 conn(0x7f7db0070870 0x7f7db00c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db50b5640 1 -- 172.21.15.136:0/314614098 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2038528683 0 0) 0x7f7dac001120 con 0x7f7db0143170 2024-04-25T11:20:30.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db50b5640 1 -- 172.21.15.136:0/314614098 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d9c003530 con 0x7f7db0143170 2024-04-25T11:20:30.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db58b6640 1 -- 172.21.15.136:0/314614098 >> v1:172.21.15.157:6789/0 conn(0x7f7db0070c80 legacy=0x7f7db0164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db58b6640 1 -- 172.21.15.136:0/314614098 >> v1:172.21.15.136:6789/0 conn(0x7f7db0143170 legacy=0x7f7db015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.609+0000 7f7db58b6640 1 -- 172.21.15.136:0/314614098 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db006fd90 con 0x7f7db0070870 2024-04-25T11:20:30.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db58b6640 1 --2- 172.21.15.136:0/314614098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0070870 0x7f7db00c6960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7da00097c0 tx=0x7f7da001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a1617c338b0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:30.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db50b5640 1 -- 172.21.15.136:0/314614098 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00b5070 con 0x7f7db0070870 2024-04-25T11:20:30.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db50b5640 1 -- 172.21.15.136:0/314614098 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da00a8990 con 0x7f7db0070870 2024-04-25T11:20:30.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db50b5640 1 -- 172.21.15.136:0/314614098 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00a8d60 con 0x7f7db0070870 2024-04-25T11:20:30.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 -- 172.21.15.136:0/314614098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0070870 msgr2=0x7f7db00c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 --2- 172.21.15.136:0/314614098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0070870 0x7f7db00c6960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7da00097c0 tx=0x7f7da001ec90 comp rx=0 tx=0).stop 2024-04-25T11:20:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 -- 172.21.15.136:0/314614098 shutdown_connections 2024-04-25T11:20:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 --2- 172.21.15.136:0/314614098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0070870 0x7f7db00c6960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 -- 172.21.15.136:0/314614098 >> 172.21.15.136:0/314614098 conn(0x7f7db00c9d50 msgr2=0x7f7db00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 -- 172.21.15.136:0/314614098 shutdown_connections 2024-04-25T11:20:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.613+0000 7f7db831b640 1 -- 172.21.15.136:0/314614098 wait complete. 2024-04-25T11:20:30.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 Processor -- start 2024-04-25T11:20:30.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- start start 2024-04-25T11:20:30.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7db0070c80 0x7f7db01f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:30.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7db00cc270 0x7f7db0152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7db015e2e0 con 0x7f7db0143170 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7db0070350 con 0x7f7db0070c80 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7db00712a0 con 0x7f7db00cc270 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55400/0 (socket says 172.21.15.136:55400) 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 -- 172.21.15.136:0/2856875961 learned_addr learned my addr 172.21.15.136:0/2856875961 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db60b7640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7db0070c80 0x7f7db01f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 -- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7db00cc270 msgr2=0x7f7db0152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7db00cc270 0x7f7db0152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 -- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7db0070c80 msgr2=0x7f7db01f4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7db0070c80 0x7f7db01f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 -- 172.21.15.136:0/2856875961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db015afa0 con 0x7f7db0143170 2024-04-25T11:20:30.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db58b6640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7da00b3040 tx=0x7f7da001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00bf070 con 0x7f7db0143170 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da00b06c0 con 0x7f7db0143170 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db015b240 con 0x7f7db0143170 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00b5040 con 0x7f7db0143170 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db015b800 con 0x7f7db0143170 2024-04-25T11:20:30.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db015c980 con 0x7f7db0143170 2024-04-25T11:20:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.617+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7da00c2070 con 0x7f7db0143170 2024-04-25T11:20:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.621+0000 7f7da6ffd640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 0x7f7d94071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.621+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7da012bd20 con 0x7f7db0143170 2024-04-25T11:20:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.621+0000 7f7db60b7640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 0x7f7d94071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.621+0000 7f7db60b7640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 0x7f7d94071940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0012c0 tx=0x7f7dac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:30.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.625+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7da00187f0 con 0x7f7db0143170 2024-04-25T11:20:30.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7db006a350 con 0x7f7db0143170 2024-04-25T11:20:30.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7da6ffd640 1 -- 172.21.15.136:0/2856875961 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6834 (secure 0 0 0) 0x7f7da00f88c0 con 0x7f7db0143170 2024-04-25T11:20:30.805 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:30.806 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4297,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:20:29.068899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4303,"qdb_cluster":[4303]},"id":1}]} 2024-04-25T11:20:30.806 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 7 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 msgr2=0x7f7d94071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 0x7f7d94071940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0012c0 tx=0x7f7dac009040 comp rx=0 tx=0).stop 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 msgr2=0x7f7db015c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7da00b3040 tx=0x7f7da001ba30 comp rx=0 tx=0).stop 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 shutdown_connections 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7d9406f430 0x7f7d94071940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7db00cc270 0x7f7db0152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7db0143170 0x7f7db015c110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 --2- 172.21.15.136:0/2856875961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7db0070c80 0x7f7db01f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 >> 172.21.15.136:0/2856875961 conn(0x7f7db00c9d50 msgr2=0x7f7db0153a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 shutdown_connections 2024-04-25T11:20:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:30.809+0000 7f7db831b640 1 -- 172.21.15.136:0/2856875961 wait complete. 2024-04-25T11:20:31.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T11:20:19.386747+0000', 'modified': '2024-04-25T11:20:29.068899+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4303}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4303': {'gid': 4303, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/1671830331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 1671830331}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 1671830331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4303, 'qdb_cluster': [4303]} 2024-04-25T11:20:31.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T11:20:31.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T11:20:31.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.593+0000 7fd7f857c640 1 Processor -- start 2024-04-25T11:20:31.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.593+0000 7fd7f857c640 1 -- start start 2024-04-25T11:20:31.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f857c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd7f0153190 0x7fd7f00c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f857c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00c8090 con 0x7fd7f00c9570 2024-04-25T11:20:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f857c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f00c6aa0 con 0x7fd7f01594b0 2024-04-25T11:20:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f857c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f00c8370 con 0x7fd7f0153190 2024-04-25T11:20:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd7f00c9570 0x7fd7f0158da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53338/0 (socket says 172.21.15.136:53338) 2024-04-25T11:20:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 -- 172.21.15.136:0/2148011865 learned_addr learned my addr 172.21.15.136:0/2148011865 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:31.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2148011865 >> v2:172.21.15.157:3300/0 conn(0x7fd7f0153190 0x7fd7f00c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878489553 0 0) 0x7fd7f00c8090 con 0x7fd7f00c9570 2024-04-25T11:20:31.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7dc005740 con 0x7fd7f00c9570 2024-04-25T11:20:31.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 339027202 0 0) 0x7fd7dc005740 con 0x7fd7f00c9570 2024-04-25T11:20:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 >> v2:172.21.15.157:3300/0 conn(0x7fd7f0153190 msgr2=0x7fd7f00c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 --2- 172.21.15.136:0/2148011865 >> v2:172.21.15.157:3300/0 conn(0x7fd7f0153190 0x7fd7f00c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 >> v1:172.21.15.157:6789/0 conn(0x7fd7f01594b0 legacy=0x7fd7f0163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f00c6d80 con 0x7fd7f00c9570 2024-04-25T11:20:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd7e0001680 con 0x7fd7f00c9570 2024-04-25T11:20:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 >> v1:172.21.15.136:6789/0 conn(0x7fd7f00c9570 legacy=0x7fd7f0158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7dc005bd0 0x7fd7dc006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7dc006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc0073e0 0x7fd7dc00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f00c8370 con 0x7fd7dc006a20 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc00e370 con 0x7fd7dc0073e0 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd7dc00e570 con 0x7fd7dc005bd0 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7dc006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc0073e0 0x7fd7dc00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7dc005bd0 0x7fd7dc006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 -- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7dc005bd0 msgr2=0x7fd7dc006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7dc005bd0 0x7fd7dc006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 -- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc0073e0 msgr2=0x7fd7dc00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc0073e0 0x7fd7dc00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f6b19640 1 -- 172.21.15.136:0/2148011865 reap_dead start 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f5b17640 1 -- 172.21.15.136:0/2148011865 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc005740 con 0x7fd7dc006a20 2024-04-25T11:20:31.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.597+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7dc005bd0 0x7fd7dc006040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:31.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7dc006e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c210 tx=0x7fd7e001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4edb6573121435 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:31.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0007d40 con 0x7fd7dc006a20 2024-04-25T11:20:31.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e00a9070 con 0x7fd7dc006a20 2024-04-25T11:20:31.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f5316640 1 -- 172.21.15.136:0/2148011865 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a5e40 con 0x7fd7dc006a20 2024-04-25T11:20:31.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 msgr2=0x7fd7dc006e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7dc006e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000c210 tx=0x7fd7e001cca0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- 172.21.15.136:0/2148011865 shutdown_connections 2024-04-25T11:20:31.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2148011865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7dc006e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- 172.21.15.136:0/2148011865 >> 172.21.15.136:0/2148011865 conn(0x7fd7f00c9d50 msgr2=0x7fd7f00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:31.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- 172.21.15.136:0/2148011865 shutdown_connections 2024-04-25T11:20:31.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- 172.21.15.136:0/2148011865 wait complete. 2024-04-25T11:20:31.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 Processor -- start 2024-04-25T11:20:31.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.601+0000 7fd7f857c640 1 -- start start 2024-04-25T11:20:31.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f00cb2c0 0x7fd7f00cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f006fae0 0x7fd7f006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f00ca870 con 0x7fd7dc006a20 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f00c7340 con 0x7fd7f00cb2c0 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f00ca9f0 con 0x7fd7f006fae0 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55410/0 (socket says 172.21.15.136:55410) 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 -- 172.21.15.136:0/2287635418 learned_addr learned my addr 172.21.15.136:0/2287635418 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f00cb2c0 0x7fd7f00cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f006fae0 0x7fd7f006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 -- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f006fae0 msgr2=0x7fd7f006fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f006fae0 0x7fd7f006fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 -- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f00cb2c0 msgr2=0x7fd7f00cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f00cb2c0 0x7fd7f00cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 -- 172.21.15.136:0/2287635418 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f01f1670 con 0x7fd7dc006a20 2024-04-25T11:20:31.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f6318640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0107c0 tx=0x7fd7ec010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7e6ffd640 1 -- 172.21.15.136:0/2287635418 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec00f040 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7e6ffd640 1 -- 172.21.15.136:0/2287635418 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec004220 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f01f1910 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7e6ffd640 1 -- 172.21.15.136:0/2287635418 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec010e70 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f01f1ed0 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7e6ffd640 1 -- 172.21.15.136:0/2287635418 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7ec00e960 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.605+0000 7fd7e6ffd640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7e6ffd640 1 -- 172.21.15.136:0/2287635418 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7ec08fae0 con 0x7fd7dc006a20 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 Processor -- start 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- start start 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 0x7fd7d4071910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00ab000 tx=0x7fd7e00b0040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 0x5600c0d18a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5600c0bca160 con 0x5600c0bfcc20 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b5fe20 con 0x5600c0be8ea0 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5600c0b18a20 con 0x5600c07e1280 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55422/0 (socket says 172.21.15.136:55422) 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 -- 172.21.15.136:0/123099832 learned_addr learned my addr 172.21.15.136:0/123099832 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 0x5600c0d18a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 -- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 msgr2=0x5600c0d18a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 0x5600c0d18a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 -- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 msgr2=0x5600c0d18550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 -- 172.21.15.136:0/123099832 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c0b1d100 con 0x5600c0bfcc20 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 0x5600c0d18a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7f6318640 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec08dcd0 tx=0x7fd7ec05bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eb3fa10b929162 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7d3fff640 1 -- 172.21.15.136:0/123099832 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec021040 con 0x5600c0bfcc20 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7d3fff640 1 -- 172.21.15.136:0/123099832 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec05c0b0 con 0x5600c0bfcc20 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7d3fff640 1 -- 172.21.15.136:0/123099832 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec060460 con 0x5600c0bfcc20 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/123099832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 msgr2=0x5600c0d4d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec08dcd0 tx=0x7fd7ec05bb30 comp rx=0 tx=0).stop 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/123099832 shutdown_connections 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c07e1280 0x5600c0d18a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/123099832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d4d5d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/123099832 >> 172.21.15.136:0/123099832 conn(0x5600c0820ff0 msgr2=0x5600c0d50030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:31.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/123099832 shutdown_connections 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.609+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/123099832 wait complete. 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- start start 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 Processor -- start 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0bfcc20 0x5600c0d578f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b6d220 con 0x5600c07e1280 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b03ea0 con 0x5600c0be8ea0 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5600c0b5ab00 con 0x5600c0bfcc20 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 10 client.0 _ll_register_callbacks cb 0x5600c0b6cdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55424/0 (socket says 172.21.15.136:55424) 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 -- 172.21.15.136:0/3670701998 learned_addr learned my addr 172.21.15.136:0/3670701998 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0bfcc20 0x5600c0d578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 -- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0bfcc20 msgr2=0x5600c0d578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0bfcc20 0x5600c0d578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 -- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 msgr2=0x5600c0d18a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 -- 172.21.15.136:0/3670701998 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c0aff500 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f6318640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec010790 tx=0x7fd7ec05e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1a1ab26736721e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec079020 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7ec05c0b0 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7ec05c520 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fd7ec021070 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 10 client.4440 fetch_fsmap learned FSMap version 7 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5600c0ae5ea0 con 0x5600c07e1280 2024-04-25T11:20:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec05c0b0 con 0x5600c07e1280 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fd7ec0649f0 con 0x5600c07e1280 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 10 client.4440 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 10 client.4440 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 20 client.4440 populate_metadata read hostname 'smithi136' 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c08057d0 0x5600c0d58630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 4 client.4440 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 --> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600c0af8be0 con 0x5600c08057d0 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.613+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c08057d0 0x5600c0d58630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.617+0000 7fd7d17fa640 10 client.4440 ms_handle_connect on v2:172.21.15.157:6838/1671830331 2024-04-25T11:20:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:31.617+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c08057d0 0x5600c0d58630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a071c43e985b67a server_cookie=36e4965b5705048e in_seq=0 out_seq=0 2024-04-25T11:20:34.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7d17fa640 1 -- 172.21.15.136:0/3670701998 <== mds.0 v2:172.21.15.157:6838/1671830331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5600c0af8be0 con 0x5600c08057d0 2024-04-25T11:20:34.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7d17fa640 10 client.4440 handle_command_reply: tid=0 2024-04-25T11:20:34.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 client.4440 shutdown 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 msgr2=0x5600c0d184e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec010790 tx=0x7fd7ec05e000 comp rx=0 tx=0).stop 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 shutdown_connections 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c08057d0 0x5600c0d58630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0bfcc20 0x5600c0d578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0be8ea0 0x5600c0d18a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/3670701998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d184e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7d17fa640 0 client.4440 ms_handle_reset on v2:172.21.15.157:6838/1671830331 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 >> 172.21.15.136:0/3670701998 conn(0x5600c0820ff0 msgr2=0x5600c0d4f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 shutdown_connections 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/3670701998 wait complete. 2024-04-25T11:20:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.081+0000 7fd7ff8d9000 20 client.4440 trim_cache size 0 max 16384 2024-04-25T11:20:34.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.113+0000 7fd7ff8d9000 1 Processor -- start 2024-04-25T11:20:34.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.113+0000 7fd7ff8d9000 1 -- start start 2024-04-25T11:20:34.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d52f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0bfcc20 0x5600c0d534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5600c0af57d0 con 0x5600c07e1280 2024-04-25T11:20:34.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b5f130 con 0x5600c0bfcc20 2024-04-25T11:20:34.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5600c0b64450 con 0x5600c0be8ea0 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55426/0 (socket says 172.21.15.136:55426) 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 -- 172.21.15.136:0/1272434687 learned_addr learned my addr 172.21.15.136:0/1272434687 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d52f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0bfcc20 0x5600c0d534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 -- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 msgr2=0x5600c0d52f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d52f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 -- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0bfcc20 msgr2=0x5600c0d534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0bfcc20 0x5600c0d534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 -- 172.21.15.136:0/1272434687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c0b18c70 con 0x5600c07e1280 2024-04-25T11:20:34.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0586a0 tx=0x7fd7ec08c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1181b0f46d427f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:34.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7d37fe640 1 -- 172.21.15.136:0/1272434687 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec079070 con 0x5600c07e1280 2024-04-25T11:20:34.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7d37fe640 1 -- 172.21.15.136:0/1272434687 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec05c0b0 con 0x5600c07e1280 2024-04-25T11:20:34.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.117+0000 7fd7d37fe640 1 -- 172.21.15.136:0/1272434687 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec060460 con 0x5600c07e1280 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1272434687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 msgr2=0x5600c0d56e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0586a0 tx=0x7fd7ec08c000 comp rx=0 tx=0).stop 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1272434687 shutdown_connections 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c0bfcc20 0x5600c0d534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d52f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1272434687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c07e1280 0x5600c0d56e70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1272434687 >> 172.21.15.136:0/1272434687 conn(0x5600c0820ff0 msgr2=0x5600c0d6c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1272434687 shutdown_connections 2024-04-25T11:20:34.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1272434687 wait complete. 2024-04-25T11:20:34.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 -- start start 2024-04-25T11:20:34.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.121+0000 7fd7ff8d9000 1 Processor -- start 2024-04-25T11:20:34.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c07e1280 0x5600c0d89a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d89fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b5ab00 con 0x5600c0bfcc20 2024-04-25T11:20:34.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5600c0b6c0c0 con 0x5600c07e1280 2024-04-25T11:20:34.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5600c0b6d220 con 0x5600c0be8ea0 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 10 client.0 _ll_register_callbacks cb 0x7fd7ec05ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55442/0 (socket says 172.21.15.136:55442) 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 -- 172.21.15.136:0/1808875646 learned_addr learned my addr 172.21.15.136:0/1808875646 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c07e1280 0x5600c0d89a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f5b17640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d89fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 -- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 msgr2=0x5600c0d89fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:34.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d89fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 -- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c07e1280 msgr2=0x5600c0d89a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:34.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c07e1280 0x5600c0d89a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 -- 172.21.15.136:0/1808875646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c0be2180 con 0x5600c0bfcc20 2024-04-25T11:20:34.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7f6b19640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00c9be0 tx=0x7fd7e8014a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6cf21f6165d418 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e801a070 con 0x5600c0bfcc20 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7ec05c0b0 con 0x5600c0bfcc20 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd7e8003b90 con 0x5600c0bfcc20 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fd7e8025050 con 0x5600c0bfcc20 2024-04-25T11:20:34.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 10 client.4446 fetch_fsmap learned FSMap version 7 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.125+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5600c08190c0 con 0x5600c0bfcc20 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec05c0b0 con 0x5600c0bfcc20 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fd7e8028020 con 0x5600c0bfcc20 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 10 client.4446 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 10 client.4446 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:20:34.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 20 client.4446 populate_metadata read hostname 'smithi136' 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c07f3830 0x5600c0dd1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 4 client.4446 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 --> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] -- command(tid 0: {"prefix": "status"}) -- 0x5600c0b6da30 con 0x5600c07f3830 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c07f3830 0x5600c0dd1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 10 client.4446 ms_handle_connect on v2:172.21.15.157:6838/1671830331 2024-04-25T11:20:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f6318640 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c07f3830 0x5600c0dd1920 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79276e7932efe024 server_cookie=7e83bc8afb916b32 in_seq=0 out_seq=0 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 1 -- 172.21.15.136:0/1808875646 <== mds.0 v2:172.21.15.157:6838/1671830331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5600c0b6da30 con 0x5600c07f3830 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 10 client.4446 handle_command_reply: tid=0 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 client.4446 shutdown 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 msgr2=0x5600c0d8a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00c9be0 tx=0x7fd7e8014a00 comp rx=0 tx=0).stop 2024-04-25T11:20:34.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 shutdown_connections 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5600c0bfcc20 0x5600c0d8a4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5600c07f3830 0x5600c0dd1920 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5600c0be8ea0 0x5600c0d89fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 --2- 172.21.15.136:0/1808875646 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5600c07e1280 0x5600c0d89a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 >> 172.21.15.136:0/1808875646 conn(0x5600c0820ff0 msgr2=0x5600c0d6cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7d17fa640 0 client.4446 ms_handle_reset on v2:172.21.15.157:6838/1671830331 2024-04-25T11:20:34.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 shutdown_connections 2024-04-25T11:20:34.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 1 -- 172.21.15.136:0/1808875646 wait complete. 2024-04-25T11:20:34.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7ff8d9000 20 client.4446 trim_cache size 0 max 16384 2024-04-25T11:20:34.139 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:20:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:20:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 0, 2024-04-25T11:20:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4303, 2024-04-25T11:20:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:20:34.140 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 14.098635495, 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 6, 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 24, 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 24, 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 21.677213432999999 2024-04-25T11:20:34.141 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 msgr2=0x7fd7d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 0x7fd7d4071910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00ab000 tx=0x7fd7e00b0040 comp rx=0 tx=0).stop 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 msgr2=0x7fd7f01499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0107c0 tx=0x7fd7ec010c80 comp rx=0 tx=0).stop 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 shutdown_connections 2024-04-25T11:20:34.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd7f006fae0 0x7fd7f006fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7f00cb2c0 0x7fd7f00cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 --2- 172.21.15.136:0/2287635418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7dc006a20 0x7fd7f01499a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 >> 172.21.15.136:0/2287635418 conn(0x7fd7f00c9d50 msgr2=0x7fd7f014cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 shutdown_connections 2024-04-25T11:20:34.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.129+0000 7fd7f857c640 1 -- 172.21.15.136:0/2287635418 wait complete. 2024-04-25T11:20:34.150 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T11:20:34.151 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:34.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T11:20:34.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 Processor -- start 2024-04-25T11:20:34.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 -- start start 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe93015a5f0 0x7fe9300c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe93015aa00 0x7fe9300c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9300c3880 con 0x7fe930154060 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe9300c8fd0 con 0x7fe93015a5f0 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe936fba640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe93006d9b0 con 0x7fe93015aa00 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe92ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe93015a5f0 0x7fe9300c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe92ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe93015a5f0 0x7fe9300c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55446/0 (socket says 172.21.15.136:55446) 2024-04-25T11:20:34.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe92ffff640 1 -- 172.21.15.136:0/2128841869 learned_addr learned my addr 172.21.15.136:0/2128841869 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:34.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.297+0000 7fe935557640 1 --2- 172.21.15.136:0/2128841869 >> v2:172.21.15.157:3300/0 conn(0x7fe93015aa00 0x7fe9300c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92ffff640 1 -- 172.21.15.136:0/2128841869 >> v2:172.21.15.157:3300/0 conn(0x7fe93015aa00 msgr2=0x7fe9300c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:34.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92ffff640 1 --2- 172.21.15.136:0/2128841869 >> v2:172.21.15.157:3300/0 conn(0x7fe93015aa00 0x7fe9300c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92ffff640 1 -- 172.21.15.136:0/2128841869 >> v1:172.21.15.157:6789/0 conn(0x7fe930154060 legacy=0x7fe9300c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92ffff640 1 -- 172.21.15.136:0/2128841869 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe930154470 con 0x7fe93015a5f0 2024-04-25T11:20:34.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92ffff640 1 --2- 172.21.15.136:0/2128841869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015a5f0 0x7fe9300c71b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe930154640 tx=0x7fe92001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79972c9d01890159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:34.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92f7fe640 1 -- 172.21.15.136:0/2128841869 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200b5070 con 0x7fe93015a5f0 2024-04-25T11:20:34.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92f7fe640 1 -- 172.21.15.136:0/2128841869 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9200a8a10 con 0x7fe93015a5f0 2024-04-25T11:20:34.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe92f7fe640 1 -- 172.21.15.136:0/2128841869 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a8de0 con 0x7fe93015a5f0 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 -- 172.21.15.136:0/2128841869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015a5f0 msgr2=0x7fe9300c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 --2- 172.21.15.136:0/2128841869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015a5f0 0x7fe9300c71b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe930154640 tx=0x7fe92001ed10 comp rx=0 tx=0).stop 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 -- 172.21.15.136:0/2128841869 shutdown_connections 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 --2- 172.21.15.136:0/2128841869 >> v2:172.21.15.157:3300/0 conn(0x7fe93015aa00 0x7fe9300c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 --2- 172.21.15.136:0/2128841869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015a5f0 0x7fe9300c71b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 -- 172.21.15.136:0/2128841869 >> 172.21.15.136:0/2128841869 conn(0x7fe9300ca5a0 msgr2=0x7fe9300ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 -- 172.21.15.136:0/2128841869 shutdown_connections 2024-04-25T11:20:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.301+0000 7fe936fba640 1 -- 172.21.15.136:0/2128841869 wait complete. 2024-04-25T11:20:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 Processor -- start 2024-04-25T11:20:34.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- start start 2024-04-25T11:20:34.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe930154060 0x7fe9301f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe93015a5f0 0x7fe9300c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe93015a0d0 con 0x7fe93015aa00 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe93006df30 con 0x7fe930154060 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe930159d80 con 0x7fe93015a5f0 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55454/0 (socket says 172.21.15.136:55454) 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 -- 172.21.15.136:0/2278921999 learned_addr learned my addr 172.21.15.136:0/2278921999 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe92ffff640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe93015a5f0 0x7fe9300c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe934d56640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe930154060 0x7fe9301f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 -- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe93015a5f0 msgr2=0x7fe9300c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe93015a5f0 0x7fe9300c5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 -- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe930154060 msgr2=0x7fe9301f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe930154060 0x7fe9301f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 -- 172.21.15.136:0/2278921999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9300c5b70 con 0x7fe93015aa00 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe935557640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe92800efc0 tx=0x7fe92800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928018020 con 0x7fe93015aa00 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe928014480 con 0x7fe93015aa00 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9300c52c0 con 0x7fe93015aa00 2024-04-25T11:20:34.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928009e60 con 0x7fe93015aa00 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.305+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301f5c10 con 0x7fe93015aa00 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe93006a350 con 0x7fe93015aa00 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe92801b020 con 0x7fe93015aa00 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe92d7fa640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 0x7fe91c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe9280908c0 con 0x7fe93015aa00 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe934d56640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 0x7fe91c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:34.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.309+0000 7fe934d56640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 0x7fe91c071910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9240015b0 tx=0x7fe924009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:34.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.313+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe928022030 con 0x7fe93015aa00 2024-04-25T11:20:34.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.493+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe9300c85e0 con 0x7fe93015aa00 2024-04-25T11:20:34.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe92d7fa640 1 -- 172.21.15.136:0/2278921999 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fe928010040 con 0x7fe93015aa00 2024-04-25T11:20:34.493 INFO:teuthology.orchestra.run.smithi136.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T11:20:34.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 msgr2=0x7fe91c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 0x7fe91c071910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9240015b0 tx=0x7fe924009040 comp rx=0 tx=0).stop 2024-04-25T11:20:34.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 msgr2=0x7fe9300709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:34.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe92800efc0 tx=0x7fe92800c490 comp rx=0 tx=0).stop 2024-04-25T11:20:34.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 shutdown_connections 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe93015aa00 0x7fe9300709b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe91c06f400 0x7fe91c071910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe93015a5f0 0x7fe9300c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 --2- 172.21.15.136:0/2278921999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe930154060 0x7fe9301f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 >> 172.21.15.136:0/2278921999 conn(0x7fe9300ca5a0 msgr2=0x7fe9300c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 shutdown_connections 2024-04-25T11:20:34.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34.497+0000 7fe936fba640 1 -- 172.21.15.136:0/2278921999 wait complete. 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi136.front.sepia.ceph.com 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T11:20:34.513 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T11:20:34.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:34.514 DEBUG:teuthology.orchestra.run.smithi136:> ip addr 2024-04-25T11:20:34.519 INFO:teuthology.orchestra.run.smithi136.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T11:20:34.519 INFO:teuthology.orchestra.run.smithi136.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T11:20:34.519 INFO:teuthology.orchestra.run.smithi136.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T11:20:34.519 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 ::1/128 scope host 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1a brd ff:ff:ff:ff:ff:ff 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f0 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1b brd ff:ff:ff:ff:ff:ff 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f1 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:42 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:43 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:20:34.520 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.21.15.136/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft 42495sec preferred_lft 42495sec 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 02:42:f7:de:2c:ef brd ff:ff:ff:ff:ff:ff 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-25T11:20:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:20:34.521 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link add name ceph-brx type bridge 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip addr flush dev ceph-brx 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set ceph-brx up 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T11:20:34.522 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:20:34.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:20:34.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:34.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:34.640 DEBUG:teuthology.orchestra.run.smithi136:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T11:20:34.692 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-04-25T11:20:34.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:34.693 DEBUG:teuthology.orchestra.run.smithi136:> route 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:Kernel IP routing table 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-25T11:20:34.755 INFO:teuthology.orchestra.run.smithi136.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-25T11:20:34.756 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T11:20:34.756 INFO:teuthology.orchestra.run.smithi136.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:20:34.756 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:20:34.756 INFO:teuthology.orchestra.run.smithi136.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:20:34.756 INFO:teuthology.orchestra.run.smithi136.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T11:20:34.756 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:20:34.756 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:20:34.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T11:20:34.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T11:20:34.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T11:20:34.757 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:20:34.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:20:34.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:34.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:34.871 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list 2024-04-25T11:20:34.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:34.917 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list-id 2024-04-25T11:20:34.965 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:20:34.965 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:20:34.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:34.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T11:20:34.965 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:20:35.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:20:35.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:35.080 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T11:20:35.080 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T11:20:35.081 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:20:35.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:20:35.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:35.284 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:20:35.284 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:20:35.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set brx.0 up 2024-04-25T11:20:35.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T11:20:35.285 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:20:35.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:20:35.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:35.397 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T11:20:35.397 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:20:35.397 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:35.444 INFO:teuthology.orchestra.run.smithi136.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T11:20:35.444 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:20:35.444 DEBUG:teuthology.orchestra.run.smithi136:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:35.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:20:35.550 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:20:35.603 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:20:35.604 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:20:35.605 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:20:35.605 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:20:35.605 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:20:35.605 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:35.605 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:35.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:35.606 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:20:35.652 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T11:20:35.652 DEBUG:teuthology.orchestra.run.smithi136:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T11:20:35.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:20:35.709 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:20:35.757 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:20:35.758 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:20:35.759 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:20:35.760 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:20:35.760 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:35.760 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:35.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:35.760 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:20:35.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi136.stderr:2024-04-25T11:20:35.809+0000 7f13d5fa8580 -1 init, newargv = 0x5646a81cd7a0 newargc=15 2024-04-25T11:20:35.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi136.stderr:ceph-fuse[33708]: starting ceph client 2024-04-25T11:20:35.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi136.stderr:ceph-fuse[33708]: starting fuse 2024-04-25T11:20:36.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:20:36.922 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:20:36.970 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:20:36.971 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:20:36.972 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:20:36.973 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:36.973 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:20:36.973 INFO:teuthology.orchestra.run.smithi136.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:36.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:36.974 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:20:37.020 INFO:teuthology.orchestra.run.smithi136.stdout:45 2024-04-25T11:20:37.021 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo stdin-killer -- python3 -c ' 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> import glob 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> import re 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> import os 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> import subprocess 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> def _find_admin_socket(client_name): 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> files = glob.glob(asok_path) 2024-04-25T11:20:37.021 DEBUG:teuthology.orchestra.run.smithi136:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> # Given a non-glob path, it better be there 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> if "*" not in asok_path: 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> assert(len(files) == 1) 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> return files[0] 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> for f in files: 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> contents = proc_f.read() 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> if mountpoint in contents: 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> return f 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T11:20:37.022 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:20:37.023 DEBUG:teuthology.orchestra.run.smithi136:> print(_find_admin_socket("client.admin")) 2024-04-25T11:20:37.023 DEBUG:teuthology.orchestra.run.smithi136:> ' 2024-04-25T11:20:37.146 INFO:teuthology.orchestra.run.smithi136.stdout:/var/run/ceph/ceph-client.admin.33708.asok 2024-04-25T11:20:37.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:37.158 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.33708.asok 2024-04-25T11:20:37.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:37.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.33708.asok status 2024-04-25T11:20:37.339 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "metadata": { 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "entity_id": "admin", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "hostname": "smithi136", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "pid": "33708", 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "root": "/" 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "dentry_count": 0, 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "dentry_pinned_count": 0, 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4461, 2024-04-25T11:20:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "inst": { 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "name": { 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "client", 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "num": 4461 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": { 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "v1", 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": "192.168.0.1:0", 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "nonce": 1155446412 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": { 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "v1", 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": "192.168.0.1:0", 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "nonce": 1155446412 2024-04-25T11:20:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "inst_str": "client.4461 192.168.0.1:0/1155446412", 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "addr_str": "192.168.0.1:0/1155446412", 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "inode_count": 1, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_epoch": 6, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_epoch": 24, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_epoch_barrier": 0, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "blocklisted": false, 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs" 2024-04-25T11:20:37.342 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:20:37.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:37.356 DEBUG:teuthology.orchestra.run.smithi136:> stat --file-system '--printf=%T 2024-04-25T11:20:37.356 DEBUG:teuthology.orchestra.run.smithi136:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:37.362 INFO:teuthology.orchestra.run.smithi136.stdout:fuseblk 2024-04-25T11:20:37.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:37.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:37.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:37.417 DEBUG:teuthology.orchestra.run.smithi136:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T11:20:37.510 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T11:20:37.547 INFO:teuthology.orchestra.run.smithi136.stdout:# file: . 2024-04-25T11:20:37.547 INFO:teuthology.orchestra.run.smithi136.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T11:20:37.547 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:37.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:20:37.555 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:20:37.555 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:20:37.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 Processor -- start 2024-04-25T11:20:37.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 -- start start 2024-04-25T11:20:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12300c4750 0x7f1230152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1230158590 0x7f12301589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f1230073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f12300710d0 con 0x7f12300c4750 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f12300c95f0 con 0x7f1230158590 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f1234955640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1230071300 con 0x7f12300736a0 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12300c4750 0x7f1230152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12300c4750 0x7f1230152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51910/0 (socket says 172.21.15.136:51910) 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 -- 172.21.15.136:0/1886367235 learned_addr learned my addr 172.21.15.136:0/1886367235 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122ed9d640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f1230073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122dd9b640 1 --2- 172.21.15.136:0/1886367235 >> v2:172.21.15.157:3300/0 conn(0x7f1230158590 0x7f12301589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 -- 172.21.15.136:0/1886367235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 msgr2=0x7f1230073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f1230073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 -- 172.21.15.136:0/1886367235 >> v2:172.21.15.157:3300/0 conn(0x7f1230158590 msgr2=0x7f12301589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 --2- 172.21.15.136:0/1886367235 >> v2:172.21.15.157:3300/0 conn(0x7f1230158590 0x7f12301589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.761+0000 7f122e59c640 1 -- 172.21.15.136:0/1886367235 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12300c7300 con 0x7f12300c4750 2024-04-25T11:20:37.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f122e59c640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f1230152900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1230071bd0 tx=0x7f122001d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33455d5950fd539 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f122d59a640 1 -- 172.21.15.136:0/1886367235 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122000b040 con 0x7f12300c4750 2024-04-25T11:20:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f122d59a640 1 -- 172.21.15.136:0/1886367235 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12200a9650 con 0x7f12300c4750 2024-04-25T11:20:37.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f122d59a640 1 -- 172.21.15.136:0/1886367235 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200a9a20 con 0x7f12300c4750 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 -- 172.21.15.136:0/1886367235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 msgr2=0x7f1230152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f1230152900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1230071bd0 tx=0x7f122001d7d0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 -- 172.21.15.136:0/1886367235 shutdown_connections 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f1230073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 --2- 172.21.15.136:0/1886367235 >> v2:172.21.15.157:3300/0 conn(0x7f1230158590 0x7f12301589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 --2- 172.21.15.136:0/1886367235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f1230152900 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 -- 172.21.15.136:0/1886367235 >> 172.21.15.136:0/1886367235 conn(0x7f12300c9d50 msgr2=0x7f12300ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:37.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 -- 172.21.15.136:0/1886367235 shutdown_connections 2024-04-25T11:20:37.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.765+0000 7f1234955640 1 -- 172.21.15.136:0/1886367235 wait complete. 2024-04-25T11:20:37.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 Processor -- start 2024-04-25T11:20:37.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- start start 2024-04-25T11:20:37.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f12301493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1230158590 0x7f1230143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f12300ca9f0 con 0x7f12300c4750 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12300ca870 con 0x7f1230158590 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12300c5720 con 0x7f12300736a0 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51926/0 (socket says 172.21.15.136:51926) 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 -- 172.21.15.136:0/4160707689 learned_addr learned my addr 172.21.15.136:0/4160707689 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122e59c640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f12301493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122ed9d640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1230158590 0x7f1230143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 -- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 msgr2=0x7f12301493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f12301493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 -- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1230158590 msgr2=0x7f1230143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:37.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1230158590 0x7f1230143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 -- 172.21.15.136:0/4160707689 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1230149900 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f122dd9b640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f122400d620 tx=0x7f122400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122400b620 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1224019070 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1230144ae0 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122400b9e0 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.769+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12301f3bc0 con 0x7f12300c4750 2024-04-25T11:20:37.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f123006a350 con 0x7f12300c4750 2024-04-25T11:20:37.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f122400bb80 con 0x7f12300c4750 2024-04-25T11:20:37.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f121effd640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 0x7f1210071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:37.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f12240902e0 con 0x7f12300c4750 2024-04-25T11:20:37.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f122e59c640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 0x7f1210071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:37.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.773+0000 7f122e59c640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 0x7f1210071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12200b0000 tx=0x7f12200a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:37.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.777+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f122405c9b0 con 0x7f12300c4750 2024-04-25T11:20:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.937+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1230070eb0 con 0x7f12300c4750 2024-04-25T11:20:37.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f121effd640 1 -- 172.21.15.136:0/4160707689 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f1224060cb0 con 0x7f12300c4750 2024-04-25T11:20:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:37.938 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 msgr2=0x7f1210071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 0x7f1210071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f12200b0000 tx=0x7f12200a8000 comp rx=0 tx=0).stop 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 msgr2=0x7f12301469f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f122400d620 tx=0x7f122400dae0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 shutdown_connections 2024-04-25T11:20:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1230158590 0x7f1230143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12300c4750 0x7f12301469f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f121006f430 0x7f1210071940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 --2- 172.21.15.136:0/4160707689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12300736a0 0x7f12301493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.941+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 >> 172.21.15.136:0/4160707689 conn(0x7f12300c9d50 msgr2=0x7f12301551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.945+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 shutdown_connections 2024-04-25T11:20:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:37.945+0000 7f1234955640 1 -- 172.21.15.136:0/4160707689 wait complete. 2024-04-25T11:20:37.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:20:37.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:20:38.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 Processor -- start 2024-04-25T11:20:38.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 -- start start 2024-04-25T11:20:38.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac406d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9ac415c480 0x7f9ac406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40b4480 con 0x7f9ac40b5810 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac40b4a90 con 0x7f9ac415c480 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9aca100640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac415c990 con 0x7f9ac415c070 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9ac40b5810 0x7f9ac40b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49884/0 (socket says 172.21.15.136:49884) 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac37fe640 1 -- 172.21.15.136:0/3086805073 learned_addr learned my addr 172.21.15.136:0/3086805073 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3086805073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac3fff640 1 --2- 172.21.15.136:0/3086805073 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415c480 0x7f9ac406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac2ffd640 1 -- 172.21.15.136:0/3086805073 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415c480 msgr2=0x7f9ac406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3086805073 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415c480 0x7f9ac406da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac2ffd640 1 -- 172.21.15.136:0/3086805073 >> v1:172.21.15.136:6789/0 conn(0x7f9ac40b5810 legacy=0x7f9ac40b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.113+0000 7f9ac2ffd640 1 -- 172.21.15.136:0/3086805073 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac415cc70 con 0x7f9ac415c070 2024-04-25T11:20:38.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.117+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3086805073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac406d510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80097c0 tx=0x7f9ab801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcb3a478f91b4b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9ac27fc640 1 -- 172.21.15.136:0/3086805073 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80b5070 con 0x7f9ac415c070 2024-04-25T11:20:38.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9ac27fc640 1 -- 172.21.15.136:0/3086805073 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab80a8990 con 0x7f9ac415c070 2024-04-25T11:20:38.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9ac27fc640 1 -- 172.21.15.136:0/3086805073 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80a8d60 con 0x7f9ac415c070 2024-04-25T11:20:38.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- 172.21.15.136:0/3086805073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 msgr2=0x7f9ac406d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 --2- 172.21.15.136:0/3086805073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac406d510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80097c0 tx=0x7f9ab801ec90 comp rx=0 tx=0).stop 2024-04-25T11:20:38.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- 172.21.15.136:0/3086805073 shutdown_connections 2024-04-25T11:20:38.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 --2- 172.21.15.136:0/3086805073 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415c480 0x7f9ac406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 --2- 172.21.15.136:0/3086805073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac406d510 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- 172.21.15.136:0/3086805073 >> 172.21.15.136:0/3086805073 conn(0x7f9ac40743a0 msgr2=0x7f9ac40d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:38.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- 172.21.15.136:0/3086805073 shutdown_connections 2024-04-25T11:20:38.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- 172.21.15.136:0/3086805073 wait complete. 2024-04-25T11:20:38.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 Processor -- start 2024-04-25T11:20:38.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.121+0000 7f9aca100640 1 -- start start 2024-04-25T11:20:38.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac4154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac415c480 0x7f9ac41531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac406e030 con 0x7f9ac40b5810 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac415cf50 con 0x7f9ac415c480 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40cbee0 con 0x7f9ac415c070 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51930/0 (socket says 172.21.15.136:51930) 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 -- 172.21.15.136:0/3379661731 learned_addr learned my addr 172.21.15.136:0/3379661731 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac4154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac3fff640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac415c480 0x7f9ac41531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 -- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 msgr2=0x7f9ac4154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac4154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 -- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac415c480 msgr2=0x7f9ac41531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac415c480 0x7f9ac41531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 -- 172.21.15.136:0/3379661731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41e5ce0 con 0x7f9ac40b5810 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac37fe640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c3d0 tx=0x7f9ab400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4017070 con 0x7f9ac40b5810 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab4005cf0 con 0x7f9ac40b5810 2024-04-25T11:20:38.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac4160070 con 0x7f9ac40b5810 2024-04-25T11:20:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40072f0 con 0x7f9ac40b5810 2024-04-25T11:20:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41f32b0 con 0x7f9ac40b5810 2024-04-25T11:20:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac41e53e0 con 0x7f9ac40b5810 2024-04-25T11:20:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9ab401a070 con 0x7f9ac40b5810 2024-04-25T11:20:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 0x7f9ab0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9ab408e850 con 0x7f9ac40b5810 2024-04-25T11:20:38.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.125+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 0x7f9ab0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.129+0000 7f9ac2ffd640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 0x7f9ab0071910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b2000 tx=0x7f9ab8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.133+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9ab405b520 con 0x7f9ac40b5810 2024-04-25T11:20:38.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.293+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ac406a350 con 0x7f9ac40b5810 2024-04-25T11:20:38.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.293+0000 7f9aabfff640 1 -- 172.21.15.136:0/3379661731 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f9ab405ae20 con 0x7f9ac40b5810 2024-04-25T11:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:20:38.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 msgr2=0x7f9ab0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 0x7f9ab0071910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b2000 tx=0x7f9ab8008040 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 msgr2=0x7f9ac41e57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c3d0 tx=0x7f9ab400c890 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 shutdown_connections 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac415c480 0x7f9ac41531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ab006f400 0x7f9ab0071910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415c070 0x7f9ac4154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 --2- 172.21.15.136:0/3379661731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40b5810 0x7f9ac41e57a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 >> 172.21.15.136:0/3379661731 conn(0x7f9ac40743a0 msgr2=0x7f9ac40c8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 shutdown_connections 2024-04-25T11:20:38.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.297+0000 7f9aca100640 1 -- 172.21.15.136:0/3379661731 wait complete. 2024-04-25T11:20:38.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:20:38.458 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:20:38.462+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T11:20:38.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 Processor -- start 2024-04-25T11:20:38.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 -- start start 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000729c0 0x7f1700142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1700072dd0 0x7f17000c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170015d590 con 0x7f17000c8190 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f170015d340 con 0x7f1700072dd0 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1707538640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f170015e1a0 con 0x7f17000729c0 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1704ad3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1700072dd0 0x7f17000c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.469+0000 7f1705ad5640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f17000c8190 0x7f1700073590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49894/0 (socket says 172.21.15.136:49894) 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1705ad5640 1 -- 172.21.15.136:0/2505172851 learned_addr learned my addr 172.21.15.136:0/2505172851 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:38.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f17052d4640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000729c0 0x7f1700142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1704ad3640 1 -- 172.21.15.136:0/2505172851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000729c0 msgr2=0x7f1700142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1704ad3640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000729c0 0x7f1700142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1704ad3640 1 -- 172.21.15.136:0/2505172851 >> v1:172.21.15.136:6789/0 conn(0x7f17000c8190 legacy=0x7f1700073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1704ad3640 1 -- 172.21.15.136:0/2505172851 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170015d870 con 0x7f1700072dd0 2024-04-25T11:20:38.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1704ad3640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1700072dd0 0x7f17000c7c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16f00097c0 tx=0x7f16f001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4aff3dd84ae19f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f16f7fff640 1 -- 172.21.15.136:0/2505172851 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f00b0070 con 0x7f1700072dd0 2024-04-25T11:20:38.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f16f7fff640 1 -- 172.21.15.136:0/2505172851 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f00ab600 con 0x7f1700072dd0 2024-04-25T11:20:38.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f16f7fff640 1 -- 172.21.15.136:0/2505172851 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f00ab9d0 con 0x7f1700072dd0 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 -- 172.21.15.136:0/2505172851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1700072dd0 msgr2=0x7f17000c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1700072dd0 0x7f17000c7c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16f00097c0 tx=0x7f16f001c330 comp rx=0 tx=0).stop 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 -- 172.21.15.136:0/2505172851 shutdown_connections 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1700072dd0 0x7f17000c7c50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 --2- 172.21.15.136:0/2505172851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000729c0 0x7f1700142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 -- 172.21.15.136:0/2505172851 >> 172.21.15.136:0/2505172851 conn(0x7f17000c9d50 msgr2=0x7f17000ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 -- 172.21.15.136:0/2505172851 shutdown_connections 2024-04-25T11:20:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.473+0000 7f1707538640 1 -- 172.21.15.136:0/2505172851 wait complete. 2024-04-25T11:20:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 Processor -- start 2024-04-25T11:20:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- start start 2024-04-25T11:20:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700072dd0 0x7f17000cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 0x7f1700138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f17000c7520 con 0x7f17000729c0 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f170015de30 con 0x7f1700072dd0 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17000c76a0 con 0x7f17000c8190 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51946/0 (socket says 172.21.15.136:51946) 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 -- 172.21.15.136:0/3555439335 learned_addr learned my addr 172.21.15.136:0/3555439335 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1705ad5640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 0x7f1700138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1704ad3640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700072dd0 0x7f17000cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 -- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 msgr2=0x7f1700138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 0x7f1700138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 -- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700072dd0 msgr2=0x7f17000cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700072dd0 0x7f17000cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 -- 172.21.15.136:0/3555439335 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17001e65c0 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1705ad5640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 0x7f1700138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f17052d4640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e8a0 tx=0x7f16fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc017070 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16fc00c940 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17001e68a0 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00ccf0 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.477+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17001e6e60 con 0x7f17000729c0 2024-04-25T11:20:38.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170006a350 con 0x7f17000729c0 2024-04-25T11:20:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f16fc01a070 con 0x7f17000729c0 2024-04-25T11:20:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f16f5ffb640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 0x7f16e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f16fc08fbc0 con 0x7f17000729c0 2024-04-25T11:20:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f1704ad3640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 0x7f16e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.481+0000 7f1704ad3640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 0x7f16e4071910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16f00ad000 tx=0x7f16f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.485+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f16fc05c8e0 con 0x7f17000729c0 2024-04-25T11:20:38.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.645+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17001544c0 con 0x7f17000729c0 2024-04-25T11:20:38.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.645+0000 7f16f5ffb640 1 -- 172.21.15.136:0/3555439335 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f16fc05c1e0 con 0x7f17000729c0 2024-04-25T11:20:38.643 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 msgr2=0x7f16e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 0x7f16e4071910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16f00ad000 tx=0x7f16f0008040 comp rx=0 tx=0).stop 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 msgr2=0x7f170015af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e8a0 tx=0x7f16fc00eca0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 shutdown_connections 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f16e406f400 0x7f16e4071910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17000c8190 0x7f1700138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700072dd0 0x7f17000cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 --2- 172.21.15.136:0/3555439335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f17000729c0 0x7f170015af10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 >> 172.21.15.136:0/3555439335 conn(0x7f17000c9d50 msgr2=0x7f170013be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:38.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 shutdown_connections 2024-04-25T11:20:38.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.649+0000 7f1707538640 1 -- 172.21.15.136:0/3555439335 wait complete. 2024-04-25T11:20:38.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:20:38.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.825+0000 7f7514c96640 1 Processor -- start 2024-04-25T11:20:38.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.825+0000 7f7514c96640 1 -- start start 2024-04-25T11:20:38.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f7514c96640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75100b3a60 0x7f7510153130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f7514c96640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f751006da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f7514c96640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75100729b0 con 0x7f75100c3460 2024-04-25T11:20:38.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f7514c96640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f75100c9df0 con 0x7f75100b3a60 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f7514c96640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75100686c0 con 0x7f751006d680 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750dd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f75100c3460 0x7f75101597f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49904/0 (socket says 172.21.15.136:49904) 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750dd9b640 1 -- 172.21.15.136:0/3083006950 learned_addr learned my addr 172.21.15.136:0/3083006950 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 --2- 172.21.15.136:0/3083006950 >> v2:172.21.15.157:3300/0 conn(0x7f75100b3a60 0x7f7510153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750ed9d640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f751006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377650984 0 0) 0x7f75100729b0 con 0x7f75100c3460 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74f8003530 con 0x7f75100c3460 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2236937285 0 0) 0x7f74f8003530 con 0x7f75100c3460 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 msgr2=0x7f751006da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f751006da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 >> v2:172.21.15.157:3300/0 conn(0x7f75100b3a60 msgr2=0x7f7510153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 --2- 172.21.15.136:0/3083006950 >> v2:172.21.15.157:3300/0 conn(0x7f75100b3a60 0x7f7510153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75100689a0 con 0x7f75100c3460 2024-04-25T11:20:38.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f75040016a0 con 0x7f75100c3460 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 >> v1:172.21.15.136:6789/0 conn(0x7f75100c3460 legacy=0x7f75101597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74f80039c0 0x7f74f8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74f8004810 0x7f74f8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f74f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f75100c9df0 con 0x7f74f80039c0 2024-04-25T11:20:38.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75100686c0 con 0x7f74f8004810 2024-04-25T11:20:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f74f800bb30 con 0x7f751006d680 2024-04-25T11:20:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74f80039c0 0x7f74f8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750dd9b640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74f8004810 0x7f74f8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750ed9d640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f74f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 -- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 msgr2=0x7f74f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f751006d680 0x7f74f8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 -- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74f8004810 msgr2=0x7f74f8004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:38.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74f8004810 0x7f74f8004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.829+0000 7f750e59c640 1 -- 172.21.15.136:0/3083006950 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f8003530 con 0x7f74f80039c0 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f750e59c640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74f80039c0 0x7f74f8003e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7500009ae0 tx=0x7f750001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6dc0559f26b149 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75000096b0 con 0x7f74f80039c0 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75000b0400 con 0x7f74f80039c0 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f750d59a640 1 -- 172.21.15.136:0/3083006950 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75000b07f0 con 0x7f74f80039c0 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- 172.21.15.136:0/3083006950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74f80039c0 msgr2=0x7f74f8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 --2- 172.21.15.136:0/3083006950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74f80039c0 0x7f74f8003e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7500009ae0 tx=0x7f750001cab0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f750ed9d640 1 -- 172.21.15.136:0/3083006950 reap_dead start 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- 172.21.15.136:0/3083006950 shutdown_connections 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- 172.21.15.136:0/3083006950 >> 172.21.15.136:0/3083006950 conn(0x7f75100ca640 msgr2=0x7f75100caa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- 172.21.15.136:0/3083006950 shutdown_connections 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- 172.21.15.136:0/3083006950 wait complete. 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 Processor -- start 2024-04-25T11:20:38.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.833+0000 7f7514c96640 1 -- start start 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 0x7f7510145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 0x7f75101549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f75100c4bc0 con 0x7f7510155720 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75100c9a90 con 0x7f7510145360 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75100c4d40 con 0x7f75101545c0 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51974/0 (socket says 172.21.15.136:51974) 2024-04-25T11:20:38.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 -- 172.21.15.136:0/3783555687 learned_addr learned my addr 172.21.15.136:0/3783555687 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750dd9b640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 0x7f75101549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750e59c640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 0x7f7510145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 -- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 msgr2=0x7f75101549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 0x7f75101549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 -- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 msgr2=0x7f7510145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 0x7f7510145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 -- 172.21.15.136:0/3783555687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7510153b20 con 0x7f7510155720 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750dd9b640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 0x7f75101549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750e59c640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 0x7f7510145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:38.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f750ed9d640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7510072ea0 tx=0x7f750800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750801c070 con 0x7f7510155720 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7508017400 con 0x7f7510155720 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7510145cb0 con 0x7f7510155720 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75080177f0 con 0x7f7510155720 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75101f2800 con 0x7f7510155720 2024-04-25T11:20:38.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f751006a350 con 0x7f7510155720 2024-04-25T11:20:38.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.837+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f750801f070 con 0x7f7510155720 2024-04-25T11:20:38.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.841+0000 7f74feffd640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 0x7f74f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:38.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.841+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f750808f9e0 con 0x7f7510155720 2024-04-25T11:20:38.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.841+0000 7f750e59c640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 0x7f74f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:38.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.841+0000 7f750e59c640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 0x7f74f0071910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75000af000 tx=0x7f75000aa040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:38.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:38.845+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7508012070 con 0x7f7510155720 2024-04-25T11:20:39.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7510159f70 con 0x7f7510155720 2024-04-25T11:20:39.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f74feffd640 1 -- 172.21.15.136:0/3783555687 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f750805c620 con 0x7f7510155720 2024-04-25T11:20:39.005 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 msgr2=0x7f74f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 0x7f74f0071910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75000af000 tx=0x7f75000aa040 comp rx=0 tx=0).stop 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 msgr2=0x7f7510155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7510072ea0 tx=0x7f750800f4a0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 shutdown_connections 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7510155720 0x7f7510155b30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f74f006f400 0x7f74f0071910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75101545c0 0x7f75101549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 --2- 172.21.15.136:0/3783555687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7510145360 0x7f7510145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 >> 172.21.15.136:0/3783555687 conn(0x7f75100ca640 msgr2=0x7f75100c2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 shutdown_connections 2024-04-25T11:20:39.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.009+0000 7f7514c96640 1 -- 172.21.15.136:0/3783555687 wait complete. 2024-04-25T11:20:39.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:20:39.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 Processor -- start 2024-04-25T11:20:39.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 -- start start 2024-04-25T11:20:39.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae00c7c50 0x7fdae00c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdae01615c0 0x7fdae01639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae0159900 con 0x7fdae00c8e40 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdae01596d0 con 0x7fdae01615c0 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae800b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdae0159bc0 con 0x7fdae00c7c50 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.185+0000 7fdae65a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdae01615c0 0x7fdae01639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdae01615c0 0x7fdae01639e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51976/0 (socket says 172.21.15.136:51976) 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 -- 172.21.15.136:0/698795706 learned_addr learned my addr 172.21.15.136:0/698795706 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae5da7640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae00c7c50 0x7fdae00c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 -- 172.21.15.136:0/698795706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae00c7c50 msgr2=0x7fdae00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae00c7c50 0x7fdae00c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 -- 172.21.15.136:0/698795706 >> v1:172.21.15.157:6789/0 conn(0x7fdae00c8e40 legacy=0x7fdae00c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 -- 172.21.15.136:0/698795706 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae015c3e0 con 0x7fdae01615c0 2024-04-25T11:20:39.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae65a8640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae01615c0 0x7fdae01639e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009e50 tx=0x7fdad801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf3479b9bf94f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae4da5640 1 -- 172.21.15.136:0/698795706 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad80b4070 con 0x7fdae01615c0 2024-04-25T11:20:39.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae4da5640 1 -- 172.21.15.136:0/698795706 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad80af400 con 0x7fdae01615c0 2024-04-25T11:20:39.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.189+0000 7fdae4da5640 1 -- 172.21.15.136:0/698795706 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad80af7d0 con 0x7fdae01615c0 2024-04-25T11:20:39.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- 172.21.15.136:0/698795706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae01615c0 msgr2=0x7fdae01639e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae01615c0 0x7fdae01639e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009e50 tx=0x7fdad801f210 comp rx=0 tx=0).stop 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- 172.21.15.136:0/698795706 shutdown_connections 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae01615c0 0x7fdae01639e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- 172.21.15.136:0/698795706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae00c7c50 0x7fdae00c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- 172.21.15.136:0/698795706 >> 172.21.15.136:0/698795706 conn(0x7fdae00c9d50 msgr2=0x7fdae00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- 172.21.15.136:0/698795706 shutdown_connections 2024-04-25T11:20:39.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- 172.21.15.136:0/698795706 wait complete. 2024-04-25T11:20:39.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 Processor -- start 2024-04-25T11:20:39.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- start start 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 0x7fdae01f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae01615c0 0x7fdae014e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdae015cc80 con 0x7fdae00c8e40 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdae015c9a0 con 0x7fdae00c7c50 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae800b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdae00c5270 con 0x7fdae01615c0 2024-04-25T11:20:39.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae55a6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae55a6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51988/0 (socket says 172.21.15.136:51988) 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae55a6640 1 -- 172.21.15.136:0/2296572858 learned_addr learned my addr 172.21.15.136:0/2296572858 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae65a8640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae01615c0 0x7fdae014e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.193+0000 7fdae5da7640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 0x7fdae01f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 -- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae01615c0 msgr2=0x7fdae014e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae01615c0 0x7fdae014e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 -- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 msgr2=0x7fdae01f1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 0x7fdae01f1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 -- 172.21.15.136:0/2296572858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae01f46d0 con 0x7fdae00c8e40 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae5da7640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 0x7fdae01f1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae55a6640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdad000c3c0 tx=0x7fdad000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad00052d0 con 0x7fdae00c8e40 2024-04-25T11:20:39.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad0005470 con 0x7fdae00c8e40 2024-04-25T11:20:39.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae0149390 con 0x7fdae00c8e40 2024-04-25T11:20:39.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0007260 con 0x7fdae00c8e40 2024-04-25T11:20:39.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae0149950 con 0x7fdae00c8e40 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae006a350 con 0x7fdae00c8e40 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdad0007400 con 0x7fdae00c8e40 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 0x7fdac4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdad008e890 con 0x7fdae00c8e40 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.197+0000 7fdae5da7640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 0x7fdac4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.201+0000 7fdae5da7640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 0x7fdac4071910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdae00c9d20 tx=0x7fdadc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.201+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdad005ae80 con 0x7fdae00c8e40 2024-04-25T11:20:39.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.357+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdae01f3db0 con 0x7fdae00c8e40 2024-04-25T11:20:39.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.357+0000 7fdad67fc640 1 -- 172.21.15.136:0/2296572858 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fdad005f180 con 0x7fdae00c8e40 2024-04-25T11:20:39.355 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:20:39.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 msgr2=0x7fdac4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 0x7fdac4071910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdae00c9d20 tx=0x7fdadc008040 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 msgr2=0x7fdae01f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdad000c3c0 tx=0x7fdad000c880 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 shutdown_connections 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdac406f400 0x7fdac4071910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdae01615c0 0x7fdae014e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdae00c8e40 0x7fdae01f4190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 --2- 172.21.15.136:0/2296572858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdae00c7c50 0x7fdae01f1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 >> 172.21.15.136:0/2296572858 conn(0x7fdae00c9d50 msgr2=0x7fdae00728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 shutdown_connections 2024-04-25T11:20:39.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.361+0000 7fdae800b640 1 -- 172.21.15.136:0/2296572858 wait complete. 2024-04-25T11:20:39.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T11:20:39.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 Processor -- start 2024-04-25T11:20:39.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 -- start start 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc0014f840 0x7fcc00074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcc00073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc000c3750 con 0x7fcc00073200 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0006dc00 con 0x7fcc0014f840 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc07952640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0014fc50 con 0x7fcc000c4340 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc04eed640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fcc00073200 0x7fcc000c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49906/0 (socket says 172.21.15.136:49906) 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc04eed640 1 -- 172.21.15.136:0/2397111990 learned_addr learned my addr 172.21.15.136:0/2397111990 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:39.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcc00073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2397111990 >> v2:172.21.15.157:3300/0 conn(0x7fcc0014f840 0x7fcc00074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765351826 0 0) 0x7fcc000c3750 con 0x7fcc00073200 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbec003530 con 0x7fcc00073200 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3973775035 0 0) 0x7fcbec003530 con 0x7fcc00073200 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 msgr2=0x7fcc00073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcc00073610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 >> v2:172.21.15.157:3300/0 conn(0x7fcc0014f840 msgr2=0x7fcc00074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 --2- 172.21.15.136:0/2397111990 >> v2:172.21.15.157:3300/0 conn(0x7fcc0014f840 0x7fcc00074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc000c3190 con 0x7fcc00073200 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fcbf0001680 con 0x7fcc00073200 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 >> v1:172.21.15.136:6789/0 conn(0x7fcc00073200 legacy=0x7fcc000c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbec0039c0 0x7fcbec003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcbec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbec004d50 0x7fcbec005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0006dc00 con 0x7fcbec0039c0 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0014fc50 con 0x7fcbec004d50 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcbec00b9d0 con 0x7fcc000c4340 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbec0039c0 0x7fcbec003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc04eed640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcbec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbec004d50 0x7fcbec005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 -- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 msgr2=0x7fcbec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc000c4340 0x7fcbec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 -- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbec004d50 msgr2=0x7fcbec005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbec004d50 0x7fcbec005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc056ee640 1 -- 172.21.15.136:0/2397111990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbec003530 con 0x7fcbec0039c0 2024-04-25T11:20:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.525+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbec004d50 0x7fcbec005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbec0039c0 0x7fcbec003e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009ae0 tx=0x7fcbfc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10727b1f5619083 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0096b0 con 0x7fcbec0039c0 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc0a6450 con 0x7fcbec0039c0 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcbf7fff640 1 -- 172.21.15.136:0/2397111990 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0a6840 con 0x7fcbec0039c0 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 -- 172.21.15.136:0/2397111990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbec0039c0 msgr2=0x7fcbec003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 --2- 172.21.15.136:0/2397111990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbec0039c0 0x7fcbec003e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009ae0 tx=0x7fcbfc01ca70 comp rx=0 tx=0).stop 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc05eef640 1 -- 172.21.15.136:0/2397111990 reap_dead start 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 -- 172.21.15.136:0/2397111990 shutdown_connections 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 -- 172.21.15.136:0/2397111990 >> 172.21.15.136:0/2397111990 conn(0x7fcc000ca5a0 msgr2=0x7fcc000ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 -- 172.21.15.136:0/2397111990 shutdown_connections 2024-04-25T11:20:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.529+0000 7fcc07952640 1 -- 172.21.15.136:0/2397111990 wait complete. 2024-04-25T11:20:39.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 Processor -- start 2024-04-25T11:20:39.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- start start 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbec0039c0 0x7fcc001f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 0x7fcc00147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00153380 con 0x7fcc00146630 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc00159c10 con 0x7fcc001587b0 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc001597f0 con 0x7fcbec0039c0 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52016/0 (socket says 172.21.15.136:52016) 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 -- 172.21.15.136:0/2737421679 learned_addr learned my addr 172.21.15.136:0/2737421679 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbec0039c0 0x7fcc001f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc04eed640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 0x7fcc00147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 -- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbec0039c0 msgr2=0x7fcc001f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbec0039c0 0x7fcc001f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 -- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 msgr2=0x7fcc00147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 0x7fcc00147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 -- 172.21.15.136:0/2737421679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc00147ac0 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc04eed640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 0x7fcc00147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc05eef640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcc001568b0 tx=0x7fcbf800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf801c070 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf8017400 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc001f37e0 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80177f0 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc001f3da0 con 0x7fcc00146630 2024-04-25T11:20:39.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.533+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0006a350 con 0x7fcc00146630 2024-04-25T11:20:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.537+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcbf801f070 con 0x7fcc00146630 2024-04-25T11:20:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.537+0000 7fcbf5ffb640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 0x7fcbe4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.537+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcbf808fe20 con 0x7fcc00146630 2024-04-25T11:20:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.537+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 0x7fcbe4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.537+0000 7fcc056ee640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 0x7fcbe4071910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0ad000 tx=0x7fcbfc0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.541+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbf8012070 con 0x7fcc00146630 2024-04-25T11:20:39.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.713+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcc0015d200 con 0x7fcc00146630 2024-04-25T11:20:39.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.713+0000 7fcbf5ffb640 1 -- 172.21.15.136:0/2737421679 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fcbf805cad0 con 0x7fcc00146630 2024-04-25T11:20:39.711 INFO:teuthology.orchestra.run.smithi136.stderr:listed 0 entries 2024-04-25T11:20:39.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 msgr2=0x7fcbe4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 0x7fcbe4071910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0ad000 tx=0x7fcbfc0046d0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 msgr2=0x7fcc0014b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcc001568b0 tx=0x7fcbf800f4a0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 shutdown_connections 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc00146630 0x7fcc0014b4e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc001587b0 0x7fcc00147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcbe406f400 0x7fcbe4071910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 --2- 172.21.15.136:0/2737421679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbec0039c0 0x7fcc001f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 >> 172.21.15.136:0/2737421679 conn(0x7fcc000ca5a0 msgr2=0x7fcc0014e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 shutdown_connections 2024-04-25T11:20:39.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.717+0000 7fcc07952640 1 -- 172.21.15.136:0/2737421679 wait complete. 2024-04-25T11:20:39.732 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:20:39.732 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:20:39.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:39.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 Processor -- start 2024-04-25T11:20:39.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 -- start start 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f40a00cdee0 0x7f40a00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a00ccb80 con 0x7f40a00b5490 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a0156740 con 0x7f40a0156330 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f40a5d9d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f40a01602f0 con 0x7f40a00cdee0 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f409ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f40a00b5490 0x7f40a00b58a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49912/0 (socket says 172.21.15.136:49912) 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f409ffff640 1 -- 172.21.15.136:0/3442838626 learned_addr learned my addr 172.21.15.136:0/3442838626 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f4096ffd640 1 --2- 172.21.15.136:0/3442838626 >> v2:172.21.15.157:3300/0 conn(0x7f40a00cdee0 0x7f40a00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609395207 0 0) 0x7f40a00ccb80 con 0x7f40a00b5490 2024-04-25T11:20:39.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.933+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f408c003530 con 0x7f40a00b5490 2024-04-25T11:20:39.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3164568627 0 0) 0x7f408c003530 con 0x7f40a00b5490 2024-04-25T11:20:39.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f4096ffd640 1 -- 172.21.15.136:0/3442838626 >> v1:172.21.15.157:6789/0 conn(0x7f40a0156330 legacy=0x7f40a015cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f4096ffd640 1 -- 172.21.15.136:0/3442838626 >> v1:172.21.15.136:6789/0 conn(0x7f40a00b5490 legacy=0x7f40a00b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f4096ffd640 1 -- 172.21.15.136:0/3442838626 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a015f6e0 con 0x7f40a00cdee0 2024-04-25T11:20:39.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f4096ffd640 1 --2- 172.21.15.136:0/3442838626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00cdee0 0x7f40a00ce2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40880097c0 tx=0x7f408801ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c06cffc3689204 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880b5070 con 0x7f40a00cdee0 2024-04-25T11:20:39.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40880a8630 con 0x7f40a00cdee0 2024-04-25T11:20:39.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.937+0000 7f409effd640 1 -- 172.21.15.136:0/3442838626 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098001a00 con 0x7f40a00cdee0 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- 172.21.15.136:0/3442838626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00cdee0 msgr2=0x7f40a00ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/3442838626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00cdee0 0x7f40a00ce2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40880097c0 tx=0x7f408801ebb0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- 172.21.15.136:0/3442838626 shutdown_connections 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/3442838626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00cdee0 0x7f40a00ce2f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- 172.21.15.136:0/3442838626 >> 172.21.15.136:0/3442838626 conn(0x7f40a006d6c0 msgr2=0x7f40a006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- 172.21.15.136:0/3442838626 shutdown_connections 2024-04-25T11:20:39.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- 172.21.15.136:0/3442838626 wait complete. 2024-04-25T11:20:39.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 Processor -- start 2024-04-25T11:20:39.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- start start 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00b5490 0x7f40a00d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40a00cdee0 0x7f40a00c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f40a015ff80 con 0x7f40a0156330 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40a015fca0 con 0x7f40a00b5490 2024-04-25T11:20:39.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.941+0000 7f40a5d9d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40a00cd320 con 0x7f40a00cdee0 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52028/0 (socket says 172.21.15.136:52028) 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 -- 172.21.15.136:0/2215401617 learned_addr learned my addr 172.21.15.136:0/2215401617 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f4096ffd640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40a00cdee0 0x7f40a00c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409f7fe640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00b5490 0x7f40a00d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 -- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40a00cdee0 msgr2=0x7f40a00c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40a00cdee0 0x7f40a00c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 -- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00b5490 msgr2=0x7f40a00d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00b5490 0x7f40a00d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 -- 172.21.15.136:0/2215401617 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a01e8020 con 0x7f40a0156330 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409ffff640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f409800cc90 tx=0x7f409800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409800ce40 con 0x7f40a0156330 2024-04-25T11:20:39.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4098012400 con 0x7f40a0156330 2024-04-25T11:20:39.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a00d20c0 con 0x7f40a0156330 2024-04-25T11:20:39.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40980127f0 con 0x7f40a0156330 2024-04-25T11:20:39.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a01f2a10 con 0x7f40a0156330 2024-04-25T11:20:39.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a006a350 con 0x7f40a0156330 2024-04-25T11:20:39.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4098012990 con 0x7f40a0156330 2024-04-25T11:20:39.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 0x7f4080071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:39.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f409808fcf0 con 0x7f40a0156330 2024-04-25T11:20:39.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.945+0000 7f409f7fe640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 0x7f4080071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:39.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.949+0000 7f409f7fe640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 0x7f4080071910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4090007fd0 tx=0x7f4090009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:39.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:39.949+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4098094070 con 0x7f40a0156330 2024-04-25T11:20:40.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.117+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40a00cd9c0 con 0x7f40a0156330 2024-04-25T11:20:40.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.117+0000 7f409cff9640 1 -- 172.21.15.136:0/2215401617 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f409805c250 con 0x7f40a0156330 2024-04-25T11:20:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:40.116 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 msgr2=0x7f4080071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 0x7f4080071910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4090007fd0 tx=0x7f4090009040 comp rx=0 tx=0).stop 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 msgr2=0x7f40a01e6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f409800cc90 tx=0x7f409800a430 comp rx=0 tx=0).stop 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 shutdown_connections 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40a0156330 0x7f40a01e6640 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f408006f400 0x7f4080071910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40a00cdee0 0x7f40a00c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 --2- 172.21.15.136:0/2215401617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40a00b5490 0x7f40a00d17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 >> 172.21.15.136:0/2215401617 conn(0x7f40a006d6c0 msgr2=0x7f40a0159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:40.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 shutdown_connections 2024-04-25T11:20:40.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.121+0000 7f40a5d9d640 1 -- 172.21.15.136:0/2215401617 wait complete. 2024-04-25T11:20:40.138 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:20:40.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:20:40.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 Processor -- start 2024-04-25T11:20:40.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 -- start start 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90a0152520 0x7f90a00736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90a00c9a40 con 0x7f90a00ca870 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f90a00c7390 con 0x7f90a0152520 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a64ef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90a00c9480 con 0x7f90a0073be0 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f9097fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90a0152520 0x7f90a00736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f909ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f90a00ca870 0x7f90a00c83c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49922/0 (socket says 172.21.15.136:49922) 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f909ffff640 1 -- 172.21.15.136:0/1715462087 learned_addr learned my addr 172.21.15.136:0/1715462087 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f90a4a8c640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f9097fff640 1 -- 172.21.15.136:0/1715462087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 msgr2=0x7f90a0162cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f9097fff640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0162cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f9097fff640 1 -- 172.21.15.136:0/1715462087 >> v1:172.21.15.136:6789/0 conn(0x7f90a00ca870 legacy=0x7f90a00c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f9097fff640 1 -- 172.21.15.136:0/1715462087 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a00c9760 con 0x7f90a0152520 2024-04-25T11:20:40.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.293+0000 7f909f7fe640 1 -- 172.21.15.136:0/1715462087 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292580069 0 0) 0x7f90a00c9a40 con 0x7f90a00ca870 2024-04-25T11:20:40.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f9097fff640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a00736a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90880097c0 tx=0x7f908801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe1cb393a46ad91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f909f7fe640 1 -- 172.21.15.136:0/1715462087 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880b0070 con 0x7f90a0152520 2024-04-25T11:20:40.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f909f7fe640 1 -- 172.21.15.136:0/1715462087 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90880ab600 con 0x7f90a0152520 2024-04-25T11:20:40.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f909f7fe640 1 -- 172.21.15.136:0/1715462087 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880ab9d0 con 0x7f90a0152520 2024-04-25T11:20:40.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 -- 172.21.15.136:0/1715462087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 msgr2=0x7f90a00736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a00736a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90880097c0 tx=0x7f908801c4a0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 -- 172.21.15.136:0/1715462087 shutdown_connections 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0162cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 --2- 172.21.15.136:0/1715462087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a00736a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 -- 172.21.15.136:0/1715462087 >> 172.21.15.136:0/1715462087 conn(0x7f90a00c9d50 msgr2=0x7f90a00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 -- 172.21.15.136:0/1715462087 shutdown_connections 2024-04-25T11:20:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.297+0000 7f90a64ef640 1 -- 172.21.15.136:0/1715462087 wait complete. 2024-04-25T11:20:40.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 Processor -- start 2024-04-25T11:20:40.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- start start 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 0x7f90a0141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f90a00c48d0 con 0x7f90a0152520 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f90a00c4750 con 0x7f90a00ca870 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90a0074250 con 0x7f90a0073be0 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52048/0 (socket says 172.21.15.136:52048) 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 -- 172.21.15.136:0/344781136 learned_addr learned my addr 172.21.15.136:0/344781136 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f9097fff640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 0x7f90a0141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f909ffff640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 -- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 msgr2=0x7f90a0144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 -- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 msgr2=0x7f90a0141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 0x7f90a0141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 -- 172.21.15.136:0/344781136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a014fa10 con 0x7f90a0152520 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f9097fff640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 0x7f90a0141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a4a8c640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9098007ce0 tx=0x7f909800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909800dba0 con 0x7f90a0152520 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f909800dd40 con 0x7f90a0152520 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a01f6360 con 0x7f90a0152520 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9098003be0 con 0x7f90a0152520 2024-04-25T11:20:40.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.301+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a01f6920 con 0x7f90a0152520 2024-04-25T11:20:40.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a01558c0 con 0x7f90a0152520 2024-04-25T11:20:40.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f909801a020 con 0x7f90a0152520 2024-04-25T11:20:40.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f909d7fa640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 0x7f9080071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f909808f550 con 0x7f90a0152520 2024-04-25T11:20:40.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f909ffff640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 0x7f9080071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.305+0000 7f909ffff640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 0x7f9080071910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f70 tx=0x7f9090007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.309+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f909805c100 con 0x7f90a0152520 2024-04-25T11:20:40.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f90a006a350 con 0x7f90a0152520 2024-04-25T11:20:40.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f909d7fa640 1 -- 172.21.15.136:0/344781136 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f909805ba00 con 0x7f90a0152520 2024-04-25T11:20:40.470 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 msgr2=0x7f9080071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 0x7f9080071910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9090000f70 tx=0x7f9090007c10 comp rx=0 tx=0).stop 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 msgr2=0x7f90a0158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9098007ce0 tx=0x7f909800d490 comp rx=0 tx=0).stop 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 shutdown_connections 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90a0152520 0x7f90a0158b80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ca870 0x7f90a0141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f908006f400 0x7f9080071910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 --2- 172.21.15.136:0/344781136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90a0073be0 0x7f90a0144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.473+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 >> 172.21.15.136:0/344781136 conn(0x7f90a00c9d50 msgr2=0x7f90a00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:40.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.477+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 shutdown_connections 2024-04-25T11:20:40.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.477+0000 7f90a64ef640 1 -- 172.21.15.136:0/344781136 wait complete. 2024-04-25T11:20:40.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:20:40.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.645+0000 7f6f9fbbf640 1 Processor -- start 2024-04-25T11:20:40.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.645+0000 7f6f9fbbf640 1 -- start start 2024-04-25T11:20:40.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9fbbf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 0x7f6f9815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9fbbf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f9806d780 0x7f6f981595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9fbbf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9806e5c0 con 0x7f6f980ca290 2024-04-25T11:20:40.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9fbbf640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f6f980c55a0 con 0x7f6f9806d780 2024-04-25T11:20:40.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9fbbf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9815c730 con 0x7f6f9806e1b0 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f9806d780 0x7f6f981595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f9806d780 0x7f6f981595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52054/0 (socket says 172.21.15.136:52054) 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 -- 172.21.15.136:0/831474253 learned_addr learned my addr 172.21.15.136:0/831474253 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d95b640 1 --2- 172.21.15.136:0/831474253 >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 0x7f6f9815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 -- 172.21.15.136:0/831474253 >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 msgr2=0x7f6f9815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 --2- 172.21.15.136:0/831474253 >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 0x7f6f9815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 -- 172.21.15.136:0/831474253 >> v1:172.21.15.157:6789/0 conn(0x7f6f980ca290 legacy=0x7f6f9815f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d15a640 1 -- 172.21.15.136:0/831474253 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9815c100 con 0x7f6f9806d780 2024-04-25T11:20:40.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.649+0000 7f6f9d95b640 1 --2- 172.21.15.136:0/831474253 >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 0x7f6f9815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:40.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9d15a640 1 --2- 172.21.15.136:0/831474253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f981595f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f980b2300 tx=0x7f6f8801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acceaabdb529955 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9c959640 1 -- 172.21.15.136:0/831474253 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f880b0070 con 0x7f6f9806d780 2024-04-25T11:20:40.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9c959640 1 -- 172.21.15.136:0/831474253 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f880ab600 con 0x7f6f9806d780 2024-04-25T11:20:40.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9c959640 1 -- 172.21.15.136:0/831474253 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f880ab9d0 con 0x7f6f9806d780 2024-04-25T11:20:40.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/831474253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 msgr2=0x7f6f981595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/831474253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f981595f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6f980b2300 tx=0x7f6f8801c450 comp rx=0 tx=0).stop 2024-04-25T11:20:40.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/831474253 shutdown_connections 2024-04-25T11:20:40.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/831474253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f981595f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/831474253 >> v2:172.21.15.157:3300/0 conn(0x7f6f9806e1b0 0x7f6f9815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/831474253 >> 172.21.15.136:0/831474253 conn(0x7f6f98070f70 msgr2=0x7f6f98068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:40.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/831474253 shutdown_connections 2024-04-25T11:20:40.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/831474253 wait complete. 2024-04-25T11:20:40.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 Processor -- start 2024-04-25T11:20:40.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.653+0000 7f6f9fbbf640 1 -- start start 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f9806e1b0 0x7f6f9814fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f980ca290 0x7f6f98150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9815c3c0 con 0x7f6f9806d780 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9806ee60 con 0x7f6f9806e1b0 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9806db90 con 0x7f6f980ca290 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52068/0 (socket says 172.21.15.136:52068) 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 -- 172.21.15.136:0/1707892097 learned_addr learned my addr 172.21.15.136:0/1707892097 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d15a640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f9806e1b0 0x7f6f9814fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9e15c640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f980ca290 0x7f6f98150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 -- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f980ca290 msgr2=0x7f6f98150380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f980ca290 0x7f6f98150380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 -- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f9806e1b0 msgr2=0x7f6f9814fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f9806e1b0 0x7f6f9814fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 -- 172.21.15.136:0/1707892097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f980cf520 con 0x7f6f9806d780 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9d95b640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400cde0 tx=0x7f6f9400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400f680 con 0x7f6f9806d780 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9400f820 con 0x7f6f9806d780 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f980cf7c0 con 0x7f6f9806d780 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400fc10 con 0x7f6f9806d780 2024-04-25T11:20:40.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f981f67b0 con 0x7f6f9806d780 2024-04-25T11:20:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.657+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9806a350 con 0x7f6f9806d780 2024-04-25T11:20:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.661+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6f94018400 con 0x7f6f9806d780 2024-04-25T11:20:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.661+0000 7f6f8e7fc640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 0x7f6f7c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.661+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6f94090640 con 0x7f6f9806d780 2024-04-25T11:20:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.661+0000 7f6f9d15a640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 0x7f6f7c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.661+0000 7f6f9d15a640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 0x7f6f7c071910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f880ad000 tx=0x7f6f88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.665+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6f94014030 con 0x7f6f9806d780 2024-04-25T11:20:40.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.825+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6f980cfdc0 con 0x7f6f9806d780 2024-04-25T11:20:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.825+0000 7f6f8e7fc640 1 -- 172.21.15.136:0/1707892097 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f6f9405d3f0 con 0x7f6f9806d780 2024-04-25T11:20:40.822 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 msgr2=0x7f6f7c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 0x7f6f7c071910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f880ad000 tx=0x7f6f88008040 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 msgr2=0x7f6f980c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400cde0 tx=0x7f6f9400a650 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 shutdown_connections 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6f7c06f400 0x7f6f7c071910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f980ca290 0x7f6f98150380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f9806e1b0 0x7f6f9814fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 --2- 172.21.15.136:0/1707892097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f9806d780 0x7f6f980c82a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 >> 172.21.15.136:0/1707892097 conn(0x7f6f98070f70 msgr2=0x7f6f9815b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:40.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 shutdown_connections 2024-04-25T11:20:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.829+0000 7f6f9fbbf640 1 -- 172.21.15.136:0/1707892097 wait complete. 2024-04-25T11:20:40.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:20:40.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 Processor -- start 2024-04-25T11:20:40.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 -- start start 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 0x7f0b540c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b540c8e40 0x7f0b540c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c4c90 0x7f0b540c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0b541596d0 con 0x7f0b540c8e40 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b54159bc0 con 0x7f0b540c7c50 2024-04-25T11:20:40.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b5b477640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b54159900 con 0x7f0b540c4c90 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b58a12640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b540c8e40 0x7f0b540c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b58a12640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b540c8e40 0x7f0b540c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52078/0 (socket says 172.21.15.136:52078) 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b58a12640 1 -- 172.21.15.136:0/2100140270 learned_addr learned my addr 172.21.15.136:0/2100140270 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b59a14640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c4c90 0x7f0b540c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:40.997+0000 7f0b59213640 1 --2- 172.21.15.136:0/2100140270 >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 0x7f0b540c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 -- 172.21.15.136:0/2100140270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c4c90 msgr2=0x7f0b540c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c4c90 0x7f0b540c68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 -- 172.21.15.136:0/2100140270 >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 msgr2=0x7f0b540c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 --2- 172.21.15.136:0/2100140270 >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 0x7f0b540c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 -- 172.21.15.136:0/2100140270 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b5415c3e0 con 0x7f0b540c8e40 2024-04-25T11:20:40.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b59213640 1 --2- 172.21.15.136:0/2100140270 >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 0x7f0b540c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:40.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b58a12640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b540c4750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b540722d0 tx=0x7f0b4401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151ba7a1c0c6b222 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:40.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b4bfff640 1 -- 172.21.15.136:0/2100140270 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440b5070 con 0x7f0b540c8e40 2024-04-25T11:20:40.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b4bfff640 1 -- 172.21.15.136:0/2100140270 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b440a89d0 con 0x7f0b540c8e40 2024-04-25T11:20:40.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b4bfff640 1 -- 172.21.15.136:0/2100140270 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440a8da0 con 0x7f0b540c8e40 2024-04-25T11:20:40.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 -- 172.21.15.136:0/2100140270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 msgr2=0x7f0b540c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:40.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b540c4750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b540722d0 tx=0x7f0b4401ed10 comp rx=0 tx=0).stop 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 -- 172.21.15.136:0/2100140270 shutdown_connections 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b540c4750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 --2- 172.21.15.136:0/2100140270 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c4c90 0x7f0b540c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 --2- 172.21.15.136:0/2100140270 >> v2:172.21.15.157:3300/0 conn(0x7f0b540c7c50 0x7f0b540c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 -- 172.21.15.136:0/2100140270 >> 172.21.15.136:0/2100140270 conn(0x7f0b540c9d50 msgr2=0x7f0b540ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 -- 172.21.15.136:0/2100140270 shutdown_connections 2024-04-25T11:20:41.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.001+0000 7f0b5b477640 1 -- 172.21.15.136:0/2100140270 wait complete. 2024-04-25T11:20:41.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 Processor -- start 2024-04-25T11:20:41.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 -- start start 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 0x7f0b541f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c7c50 0x7f0b541f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5415cc80 con 0x7f0b540c8e40 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5415c9a0 con 0x7f0b540c4c90 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b540c5270 con 0x7f0b540c7c50 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52086/0 (socket says 172.21.15.136:52086) 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 -- 172.21.15.136:0/3843221712 learned_addr learned my addr 172.21.15.136:0/3843221712 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59213640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 0x7f0b541f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b58a12640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c7c50 0x7f0b541f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 -- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c7c50 msgr2=0x7f0b541f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c7c50 0x7f0b541f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 -- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 msgr2=0x7f0b541f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 0x7f0b541f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 -- 172.21.15.136:0/3843221712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b541f18b0 con 0x7f0b540c8e40 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59213640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 0x7f0b541f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b59a14640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00eed0 tx=0x7f0b4c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c018020 con 0x7f0b540c8e40 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b4c00bd10 con 0x7f0b540c8e40 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b540c6df0 con 0x7f0b540c8e40 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.005+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c014640 con 0x7f0b540c8e40 2024-04-25T11:20:41.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b541f0890 con 0x7f0b540c8e40 2024-04-25T11:20:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b541f3100 con 0x7f0b540c8e40 2024-04-25T11:20:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0b4c01b020 con 0x7f0b540c8e40 2024-04-25T11:20:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b49ffb640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 0x7f0b38071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0b4c08fca0 con 0x7f0b540c8e40 2024-04-25T11:20:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b59213640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 0x7f0b38071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.009+0000 7f0b59213640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 0x7f0b38071940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b540c9d20 tx=0x7f0b50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.013+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0b4c022030 con 0x7f0b540c8e40 2024-04-25T11:20:41.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0b5406a350 con 0x7f0b540c8e40 2024-04-25T11:20:41.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b49ffb640 1 -- 172.21.15.136:0/3843221712 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f0b4c05c970 con 0x7f0b540c8e40 2024-04-25T11:20:41.166 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:20:41.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 msgr2=0x7f0b38071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 0x7f0b38071940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b540c9d20 tx=0x7f0b50008040 comp rx=0 tx=0).stop 2024-04-25T11:20:41.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 msgr2=0x7f0b5414eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00eed0 tx=0x7f0b4c00b480 comp rx=0 tx=0).stop 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 shutdown_connections 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b540c8e40 0x7f0b5414eba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0b3806f430 0x7f0b38071940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b540c7c50 0x7f0b541f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 --2- 172.21.15.136:0/3843221712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b540c4c90 0x7f0b541f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 >> 172.21.15.136:0/3843221712 conn(0x7f0b540c9d50 msgr2=0x7f0b54157e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 shutdown_connections 2024-04-25T11:20:41.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.169+0000 7f0b5b477640 1 -- 172.21.15.136:0/3843221712 wait complete. 2024-04-25T11:20:41.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:20:41.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 Processor -- start 2024-04-25T11:20:41.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 -- start start 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff43415cbc0 0x7ff43406d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43406e5c0 con 0x7ff43406e1b0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43415cfd0 con 0x7ff4340ca290 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4395cd640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff4340c55a0 con 0x7ff43415cbc0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff432ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff43406e1b0 0x7ff4341595f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49938/0 (socket says 172.21.15.136:49938) 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff432ffd640 1 -- 172.21.15.136:0/4283505203 learned_addr learned my addr 172.21.15.136:0/4283505203 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4327fc640 1 --2- 172.21.15.136:0/4283505203 >> v2:172.21.15.136:3300/0 conn(0x7ff43415cbc0 0x7ff43406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340240392 0 0) 0x7ff43406e5c0 con 0x7ff43406e1b0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff41c003530 con 0x7ff43406e1b0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 419145920 0 0) 0x7ff41c003530 con 0x7ff43406e1b0 2024-04-25T11:20:41.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4327fc640 1 -- 172.21.15.136:0/4283505203 >> v1:172.21.15.157:6789/0 conn(0x7ff4340ca290 legacy=0x7ff4341669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4327fc640 1 -- 172.21.15.136:0/4283505203 >> v1:172.21.15.136:6789/0 conn(0x7ff43406e1b0 legacy=0x7ff4341595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.345+0000 7ff4327fc640 1 -- 172.21.15.136:0/4283505203 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43415c100 con 0x7ff43415cbc0 2024-04-25T11:20:41.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4327fc640 1 --2- 172.21.15.136:0/4283505203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43415cbc0 0x7ff43406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4280097c0 tx=0x7ff42801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3abe244e47af93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4280b0070 con 0x7ff43415cbc0 2024-04-25T11:20:41.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4280ab400 con 0x7ff43415cbc0 2024-04-25T11:20:41.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff431ffb640 1 -- 172.21.15.136:0/4283505203 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240016a0 con 0x7ff43415cbc0 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 -- 172.21.15.136:0/4283505203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43415cbc0 msgr2=0x7ff43406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 --2- 172.21.15.136:0/4283505203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43415cbc0 0x7ff43406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4280097c0 tx=0x7ff42801c330 comp rx=0 tx=0).stop 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 -- 172.21.15.136:0/4283505203 shutdown_connections 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 --2- 172.21.15.136:0/4283505203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43415cbc0 0x7ff43406d780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 -- 172.21.15.136:0/4283505203 >> 172.21.15.136:0/4283505203 conn(0x7ff434070f70 msgr2=0x7ff434068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 -- 172.21.15.136:0/4283505203 shutdown_connections 2024-04-25T11:20:41.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.349+0000 7ff4395cd640 1 -- 172.21.15.136:0/4283505203 wait complete. 2024-04-25T11:20:41.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 Processor -- start 2024-04-25T11:20:41.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- start start 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4340ca290 0x7ff4340b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 0x7ff4341ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff43415c3c0 con 0x7ff43406e1b0 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff43415c730 con 0x7ff4340ca290 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4340caab0 con 0x7ff4341f5c90 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52108/0 (socket says 172.21.15.136:52108) 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 -- 172.21.15.136:0/1250959705 learned_addr learned my addr 172.21.15.136:0/1250959705 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4327fc640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4340ca290 0x7ff4340b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4337fe640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 0x7ff4341ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 -- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 msgr2=0x7ff4341ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 0x7ff4341ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 -- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4340ca290 msgr2=0x7ff4340b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4340ca290 0x7ff4340b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 -- 172.21.15.136:0/1250959705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4340c9330 con 0x7ff43406e1b0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4337fe640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 0x7ff4341ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff432ffd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff42400c530 tx=0x7ff42400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424017070 con 0x7ff43406e1b0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff424005cf0 con 0x7ff43406e1b0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341f9cb0 con 0x7ff43406e1b0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424007290 con 0x7ff43406e1b0 2024-04-25T11:20:41.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341f9f70 con 0x7ff43406e1b0 2024-04-25T11:20:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.353+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff43406a350 con 0x7ff43406e1b0 2024-04-25T11:20:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.357+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff42401a070 con 0x7ff43406e1b0 2024-04-25T11:20:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.357+0000 7ff41b7fe640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.357+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff42408e840 con 0x7ff43406e1b0 2024-04-25T11:20:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.357+0000 7ff4327fc640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.357+0000 7ff4327fc640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff4280ad000 tx=0x7ff4280a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.361+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff42405ae30 con 0x7ff43406e1b0 2024-04-25T11:20:41.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.525+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff4340cfdc0 con 0x7ff43406e1b0 2024-04-25T11:20:41.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.525+0000 7ff41b7fe640 1 -- 172.21.15.136:0/1250959705 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7ff42405f130 con 0x7ff43406e1b0 2024-04-25T11:20:41.522 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:20:41.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 msgr2=0x7ff420071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff4280ad000 tx=0x7ff4280a6000 comp rx=0 tx=0).stop 2024-04-25T11:20:41.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 msgr2=0x7ff4340b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff42400c530 tx=0x7ff42400c9f0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 shutdown_connections 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4341f5c90 0x7ff4341ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4340ca290 0x7ff4340b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 --2- 172.21.15.136:0/1250959705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43406e1b0 0x7ff4340b46f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 >> 172.21.15.136:0/1250959705 conn(0x7ff434070f70 msgr2=0x7ff43415a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 shutdown_connections 2024-04-25T11:20:41.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.529+0000 7ff4395cd640 1 -- 172.21.15.136:0/1250959705 wait complete. 2024-04-25T11:20:41.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T11:20:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 Processor -- start 2024-04-25T11:20:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 -- start start 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facb80c96e0 0x7facb81540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7facb80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb80c9af0 con 0x7facb80c79e0 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7facb806cc10 con 0x7facb80c96e0 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbf497640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7facb80cb220 con 0x7facb815aae0 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbca32640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7facb80c79e0 0x7facb815a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49942/0 (socket says 172.21.15.136:49942) 2024-04-25T11:20:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbca32640 1 -- 172.21.15.136:0/1477869649 learned_addr learned my addr 172.21.15.136:0/1477869649 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:41.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbda34640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7facb80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facbd233640 1 --2- 172.21.15.136:0/1477869649 >> v2:172.21.15.157:3300/0 conn(0x7facb80c96e0 0x7facb81540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821140598 0 0) 0x7facb80c9af0 con 0x7facb80c79e0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca0003530 con 0x7facb80c79e0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1082490884 0 0) 0x7faca0003530 con 0x7facb80c79e0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 msgr2=0x7facb80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7facb80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 >> v2:172.21.15.157:3300/0 conn(0x7facb80c96e0 msgr2=0x7facb81540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 --2- 172.21.15.136:0/1477869649 >> v2:172.21.15.157:3300/0 conn(0x7facb80c96e0 0x7facb81540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb80cb500 con 0x7facb80c79e0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.693+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7facb4001a20 con 0x7facb80c79e0 2024-04-25T11:20:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 >> v1:172.21.15.136:6789/0 conn(0x7facb80c79e0 legacy=0x7facb815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faca00039c0 0x7faca0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7faca0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faca0004d50 0x7faca0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7facb806cc10 con 0x7faca0004d50 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facb80cb220 con 0x7faca00039c0 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faca000ba50 con 0x7facb815aae0 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faca0004d50 0x7faca0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbca32640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7faca0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbd233640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faca00039c0 0x7faca0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 -- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 msgr2=0x7faca0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb815aae0 0x7faca0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 -- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faca00039c0 msgr2=0x7faca0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faca00039c0 0x7faca0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 -- 172.21.15.136:0/1477869649 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca0003530 con 0x7faca0004d50 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faca0004d50 0x7faca0005160 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7facb8075160 tx=0x7facb001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474e8e984fae4fad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb00b2070 con 0x7faca0004d50 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb00a5620 con 0x7faca0004d50 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facaffff640 1 -- 172.21.15.136:0/1477869649 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb00096b0 con 0x7faca0004d50 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 -- 172.21.15.136:0/1477869649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faca0004d50 msgr2=0x7faca0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 --2- 172.21.15.136:0/1477869649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faca0004d50 0x7faca0005160 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7facb8075160 tx=0x7facb001ca30 comp rx=0 tx=0).stop 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbda34640 1 -- 172.21.15.136:0/1477869649 reap_dead start 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 -- 172.21.15.136:0/1477869649 shutdown_connections 2024-04-25T11:20:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 -- 172.21.15.136:0/1477869649 >> 172.21.15.136:0/1477869649 conn(0x7facb806f9d0 msgr2=0x7facb806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 -- 172.21.15.136:0/1477869649 shutdown_connections 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.697+0000 7facbf497640 1 -- 172.21.15.136:0/1477869649 wait complete. 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 Processor -- start 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- start start 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 0x7facb81e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb8149d60 0x7facb81f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7facb80cf0f0 con 0x7facb8149950 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facb8154610 con 0x7facb81e5d20 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7facb80ced40 con 0x7facb8149d60 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52122/0 (socket says 172.21.15.136:52122) 2024-04-25T11:20:41.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 -- 172.21.15.136:0/3734157193 learned_addr learned my addr 172.21.15.136:0/3734157193 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbd233640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 0x7facb81e33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbda34640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb8149d60 0x7facb81f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 -- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb8149d60 msgr2=0x7facb81f04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb8149d60 0x7facb81f04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 -- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 msgr2=0x7facb81e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 0x7facb81e33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 -- 172.21.15.136:0/3734157193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb81e3a80 con 0x7facb8149950 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbd233640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 0x7facb81e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbca32640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7facb400e410 tx=0x7facb400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4012650 con 0x7facb8149950 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb4018070 con 0x7facb8149950 2024-04-25T11:20:41.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb81f36c0 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4012a30 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.701+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb81f3c80 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb806a350 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7facb4012bd0 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facadffb640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 0x7faca4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7facb4090240 con 0x7facb8149950 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facbd233640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 0x7faca4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:41.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.705+0000 7facbd233640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 0x7faca4071910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faca8009a20 tx=0x7faca8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:41.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.709+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facb405cdf0 con 0x7facb8149950 2024-04-25T11:20:41.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.877+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7facb806a8d0 con 0x7facb8149950 2024-04-25T11:20:41.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.877+0000 7facadffb640 1 -- 172.21.15.136:0/3734157193 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7facb4063e20 con 0x7facb8149950 2024-04-25T11:20:41.874 INFO:teuthology.orchestra.run.smithi136.stderr:listed 0 entries 2024-04-25T11:20:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 msgr2=0x7faca4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 0x7faca4071910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faca8009a20 tx=0x7faca8008040 comp rx=0 tx=0).stop 2024-04-25T11:20:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 msgr2=0x7facb81effb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7facb400e410 tx=0x7facb400e8d0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 shutdown_connections 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faca406f400 0x7faca4071910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facb8149d60 0x7facb81f04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facb8149950 0x7facb81effb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 --2- 172.21.15.136:0/3734157193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facb81e5d20 0x7facb81e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 >> 172.21.15.136:0/3734157193 conn(0x7facb806f9d0 msgr2=0x7facb806df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 shutdown_connections 2024-04-25T11:20:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:41.881+0000 7facbf497640 1 -- 172.21.15.136:0/3734157193 wait complete. 2024-04-25T11:20:41.895 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi136.front.sepia.ceph.com... 2024-04-25T11:20:41.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:41.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T11:20:41.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi136.stderr:ceph-fuse[33708]: fuse finished with error 0 and tester_r 0 2024-04-25T11:20:41.989 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:20:48.090 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:20:48.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:20:48.090 DEBUG:teuthology.orchestra.run.smithi136:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T11:20:48.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:48.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 Processor -- start 2024-04-25T11:20:48.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 -- start start 2024-04-25T11:20:48.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2f64153190 0x7f2f640c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f641596f0 con 0x7f2f641592e0 2024-04-25T11:20:48.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2f640c8370 con 0x7f2f64153190 2024-04-25T11:20:48.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f6bf17640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2f640c8090 con 0x7f2f640c9570 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f694b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f69cb3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2f64153190 0x7f2f640c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f694b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:51798/0 (socket says 172.21.15.136:51798) 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f694b2640 1 -- 172.21.15.136:0/672432796 learned_addr learned my addr 172.21.15.136:0/672432796 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f69cb3640 1 -- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 msgr2=0x7f2f64158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f69cb3640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f69cb3640 1 -- 172.21.15.136:0/672432796 >> v1:172.21.15.157:6789/0 conn(0x7f2f641592e0 legacy=0x7f2f6415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f69cb3640 1 -- 172.21.15.136:0/672432796 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f640c6aa0 con 0x7f2f64153190 2024-04-25T11:20:48.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.304+0000 7f2f694b2640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f69cb3640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f64153190 0x7f2f640c9030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097c0 tx=0x7f2f6001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fca829ad95f3eb99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f68cb1640 1 -- 172.21.15.136:0/672432796 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600b5070 con 0x7f2f64153190 2024-04-25T11:20:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f68cb1640 1 -- 172.21.15.136:0/672432796 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f600a8990 con 0x7f2f64153190 2024-04-25T11:20:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f68cb1640 1 -- 172.21.15.136:0/672432796 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600a8d60 con 0x7f2f64153190 2024-04-25T11:20:48.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f6bf17640 1 -- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f64153190 msgr2=0x7f2f640c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.308+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f64153190 0x7f2f640c9030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f600097c0 tx=0x7f2f6001ec90 comp rx=0 tx=0).stop 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- 172.21.15.136:0/672432796 shutdown_connections 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f64158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/672432796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f64153190 0x7f2f640c9030 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- 172.21.15.136:0/672432796 >> 172.21.15.136:0/672432796 conn(0x7f2f640c9d50 msgr2=0x7f2f640ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- 172.21.15.136:0/672432796 shutdown_connections 2024-04-25T11:20:48.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- 172.21.15.136:0/672432796 wait complete. 2024-04-25T11:20:48.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 Processor -- start 2024-04-25T11:20:48.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- start start 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f6415ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 0x7f2f641f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2f640c7340 con 0x7f2f64153190 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2f640c7060 con 0x7f2f641592e0 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6bf17640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2f640ca870 con 0x7f2f640c9570 2024-04-25T11:20:48.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36828/0 (socket says 172.21.15.136:36828) 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 -- 172.21.15.136:0/2994563503 learned_addr learned my addr 172.21.15.136:0/2994563503 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6a4b4640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 0x7f2f641f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f69cb3640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f6415ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 -- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 msgr2=0x7f2f6415ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f6415ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 -- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 msgr2=0x7f2f641f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 0x7f2f641f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f694b2640 1 -- 172.21.15.136:0/2994563503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6415cfc0 con 0x7f2f64153190 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.312+0000 7f2f6a4b4640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 0x7f2f641f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f694b2640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800e960 tx=0x7f2f5800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58017070 con 0x7f2f64153190 2024-04-25T11:20:48.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f5800ca00 con 0x7f2f64153190 2024-04-25T11:20:48.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6415a450 con 0x7f2f64153190 2024-04-25T11:20:48.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5800cdb0 con 0x7f2f64153190 2024-04-25T11:20:48.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f6415a9f0 con 0x7f2f64153190 2024-04-25T11:20:48.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f6406a350 con 0x7f2f64153190 2024-04-25T11:20:48.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2f5801a070 con 0x7f2f64153190 2024-04-25T11:20:48.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 0x7f2f48071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2f5808f470 con 0x7f2f64153190 2024-04-25T11:20:48.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f69cb3640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 0x7f2f48071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.316+0000 7f2f69cb3640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 0x7f2f48071940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f600b2000 tx=0x7f2f60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.320+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2f5805c030 con 0x7f2f64153190 2024-04-25T11:20:48.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.484+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f641f4b50 con 0x7f2f64153190 2024-04-25T11:20:48.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f567fc640 1 -- 172.21.15.136:0/2994563503 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f2f5805b930 con 0x7f2f64153190 2024-04-25T11:20:48.486 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:48.486 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 msgr2=0x7f2f48071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 0x7f2f48071940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f600b2000 tx=0x7f2f60008040 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 msgr2=0x7f2f641f3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800e960 tx=0x7f2f5800ee20 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 shutdown_connections 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2f641592e0 0x7f2f641f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f64153190 0x7f2f641f3fa0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2f4806f430 0x7f2f48071940 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 --2- 172.21.15.136:0/2994563503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2f640c9570 0x7f2f6415ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 >> 172.21.15.136:0/2994563503 conn(0x7f2f640c9d50 msgr2=0x7f2f6406fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:48.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 shutdown_connections 2024-04-25T11:20:48.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.488+0000 7f2f6bf17640 1 -- 172.21.15.136:0/2994563503 wait complete. 2024-04-25T11:20:48.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:48.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.640+0000 7f1dae1f2640 1 Processor -- start 2024-04-25T11:20:48.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 -- start start 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1da8143170 0x7f1da815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1da8070c80 0x7f1da8164790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da8143580 con 0x7f1da8070870 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1da815dd20 con 0x7f1da8070c80 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1dae1f2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1da815e000 con 0x7f1da8143170 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1da8070c80 0x7f1da8164790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1da8070c80 0x7f1da8164790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36842/0 (socket says 172.21.15.136:36842) 2024-04-25T11:20:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 -- 172.21.15.136:0/2939821221 learned_addr learned my addr 172.21.15.136:0/2939821221 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da77fe640 1 --2- 172.21.15.136:0/2939821221 >> v2:172.21.15.157:3300/0 conn(0x7f1da8143170 0x7f1da815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 -- 172.21.15.136:0/2939821221 >> v2:172.21.15.157:3300/0 conn(0x7f1da8143170 msgr2=0x7f1da815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 --2- 172.21.15.136:0/2939821221 >> v2:172.21.15.157:3300/0 conn(0x7f1da8143170 0x7f1da815d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 -- 172.21.15.136:0/2939821221 >> v1:172.21.15.157:6789/0 conn(0x7f1da8070870 legacy=0x7f1da80c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.644+0000 7f1da7fff640 1 -- 172.21.15.136:0/2939821221 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da806fd90 con 0x7f1da8070c80 2024-04-25T11:20:48.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1da7fff640 1 --2- 172.21.15.136:0/2939821221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8070c80 0x7f1da8164790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1da80c9cb0 tx=0x7f1d9801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af92ef68f0feb0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1da67fc640 1 -- 172.21.15.136:0/2939821221 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980b4070 con 0x7f1da8070c80 2024-04-25T11:20:48.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1da67fc640 1 -- 172.21.15.136:0/2939821221 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980af400 con 0x7f1da8070c80 2024-04-25T11:20:48.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1da67fc640 1 -- 172.21.15.136:0/2939821221 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980af7d0 con 0x7f1da8070c80 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- 172.21.15.136:0/2939821221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8070c80 msgr2=0x7f1da8164790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/2939821221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8070c80 0x7f1da8164790 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1da80c9cb0 tx=0x7f1d9801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- 172.21.15.136:0/2939821221 shutdown_connections 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/2939821221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8070c80 0x7f1da8164790 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/2939821221 >> v2:172.21.15.157:3300/0 conn(0x7f1da8143170 0x7f1da815d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- 172.21.15.136:0/2939821221 >> 172.21.15.136:0/2939821221 conn(0x7f1da80c9d50 msgr2=0x7f1da80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- 172.21.15.136:0/2939821221 shutdown_connections 2024-04-25T11:20:48.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- 172.21.15.136:0/2939821221 wait complete. 2024-04-25T11:20:48.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 Processor -- start 2024-04-25T11:20:48.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.648+0000 7f1dae1f2640 1 -- start start 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 0x7f1da8158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 0x7f1da815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1da815e2e0 con 0x7f1da8143170 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1da8070350 con 0x7f1da8070870 2024-04-25T11:20:48.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1da80712a0 con 0x7f1da8070c80 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36846/0 (socket says 172.21.15.136:36846) 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 -- 172.21.15.136:0/667924263 learned_addr learned my addr 172.21.15.136:0/667924263 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da6ffd640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 0x7f1da815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da77fe640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 0x7f1da8158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 -- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 msgr2=0x7f1da815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 0x7f1da815ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 -- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 msgr2=0x7f1da8158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 0x7f1da8158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 -- 172.21.15.136:0/667924263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da8158770 con 0x7f1da8143170 2024-04-25T11:20:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da6ffd640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 0x7f1da815ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da77fe640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 0x7f1da8158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1da7fff640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d9801ce10 tx=0x7f1d9801bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980c0070 con 0x7f1da8143170 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980090d0 con 0x7f1da8143170 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980b4040 con 0x7f1da8143170 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da8159160 con 0x7f1da8143170 2024-04-25T11:20:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da815c020 con 0x7f1da8143170 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da80cb180 con 0x7f1da8143170 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.652+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1d980c3070 con 0x7f1da8143170 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.656+0000 7f1d9ffff640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 0x7f1d88071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.656+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1d9812cc60 con 0x7f1da8143170 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.656+0000 7f1da77fe640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 0x7f1d88071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.656+0000 7f1da77fe640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 0x7f1d88071910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097a0 tx=0x7f1da0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.660+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1d98018d40 con 0x7f1da8143170 2024-04-25T11:20:48.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.816+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1da806a350 con 0x7f1da8143170 2024-04-25T11:20:48.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.816+0000 7f1d9ffff640 1 -- 172.21.15.136:0/667924263 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f1d980f9810 con 0x7f1da8143170 2024-04-25T11:20:48.816 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:48.816 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:48.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 msgr2=0x7f1d88071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 0x7f1d88071910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1da00097a0 tx=0x7f1da0008040 comp rx=0 tx=0).stop 2024-04-25T11:20:48.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 msgr2=0x7f1da813cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d9801ce10 tx=0x7f1d9801bf80 comp rx=0 tx=0).stop 2024-04-25T11:20:48.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 shutdown_connections 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1da8143170 0x7f1da813cc40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d8806f400 0x7f1d88071910 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1da8070c80 0x7f1da815ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 --2- 172.21.15.136:0/667924263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1da8070870 0x7f1da8158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 >> 172.21.15.136:0/667924263 conn(0x7f1da80c9d50 msgr2=0x7f1da80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 shutdown_connections 2024-04-25T11:20:48.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.820+0000 7f1dae1f2640 1 -- 172.21.15.136:0/667924263 wait complete. 2024-04-25T11:20:48.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T11:20:48.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 Processor -- start 2024-04-25T11:20:48.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 -- start start 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 0x7f1538142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1538072dd0 0x7f15380c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153815d590 con 0x7f15380c8190 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f153815d340 con 0x7f1538072dd0 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153d064640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f153815e1a0 con 0x7f15380729c0 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1538072dd0 0x7f15380c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153759e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f15380c8190 0x7f1538073590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53264/0 (socket says 172.21.15.136:53264) 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153759e640 1 -- 172.21.15.136:0/2454054517 learned_addr learned my addr 172.21.15.136:0/2454054517 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1538072dd0 0x7f15380c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36856/0 (socket says 172.21.15.136:36856) 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f1536d9d640 1 --2- 172.21.15.136:0/2454054517 >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 0x7f1538142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 -- 172.21.15.136:0/2454054517 >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 msgr2=0x7f1538142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 --2- 172.21.15.136:0/2454054517 >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 0x7f1538142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 -- 172.21.15.136:0/2454054517 >> v1:172.21.15.136:6789/0 conn(0x7f15380c8190 legacy=0x7f1538073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f1535d9b640 1 -- 172.21.15.136:0/2454054517 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542521657 0 0) 0x7f153815d590 con 0x7f15380c8190 2024-04-25T11:20:48.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f153659c640 1 -- 172.21.15.136:0/2454054517 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153815d870 con 0x7f1538072dd0 2024-04-25T11:20:48.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.988+0000 7f1536d9d640 1 --2- 172.21.15.136:0/2454054517 >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 0x7f1538142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:48.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153659c640 1 --2- 172.21.15.136:0/2454054517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1538072dd0 0x7f15380c7c50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f15380c9cb0 tx=0x7f152c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f179f4393543734 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f1535d9b640 1 -- 172.21.15.136:0/2454054517 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c0b0070 con 0x7f1538072dd0 2024-04-25T11:20:48.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f1535d9b640 1 -- 172.21.15.136:0/2454054517 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f152c0ab600 con 0x7f1538072dd0 2024-04-25T11:20:48.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f1535d9b640 1 -- 172.21.15.136:0/2454054517 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c0ab9d0 con 0x7f1538072dd0 2024-04-25T11:20:48.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 -- 172.21.15.136:0/2454054517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1538072dd0 msgr2=0x7f15380c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 --2- 172.21.15.136:0/2454054517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1538072dd0 0x7f15380c7c50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f15380c9cb0 tx=0x7f152c01c480 comp rx=0 tx=0).stop 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 -- 172.21.15.136:0/2454054517 shutdown_connections 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 --2- 172.21.15.136:0/2454054517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1538072dd0 0x7f15380c7c50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 --2- 172.21.15.136:0/2454054517 >> v2:172.21.15.157:3300/0 conn(0x7f15380729c0 0x7f1538142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 -- 172.21.15.136:0/2454054517 >> 172.21.15.136:0/2454054517 conn(0x7f15380c9d50 msgr2=0x7f15380ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 -- 172.21.15.136:0/2454054517 shutdown_connections 2024-04-25T11:20:48.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.992+0000 7f153d064640 1 -- 172.21.15.136:0/2454054517 wait complete. 2024-04-25T11:20:48.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 Processor -- start 2024-04-25T11:20:48.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- start start 2024-04-25T11:20:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15380729c0 0x7f1538139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1538072dd0 0x7f1538152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1538143370 con 0x7f15380c8190 2024-04-25T11:20:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f15381431f0 con 0x7f15380729c0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1538166ef0 con 0x7f1538072dd0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36864/0 (socket says 172.21.15.136:36864) 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 -- 172.21.15.136:0/1578713336 learned_addr learned my addr 172.21.15.136:0/1578713336 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f1536d9d640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15380729c0 0x7f1538139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153659c640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1538072dd0 0x7f1538152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 -- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1538072dd0 msgr2=0x7f1538152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1538072dd0 0x7f1538152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 -- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15380729c0 msgr2=0x7f1538139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15380729c0 0x7f1538139c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 -- 172.21.15.136:0/1578713336 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1538133540 con 0x7f15380c8190 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153759e640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f153000c6d0 tx=0x7f153000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:48.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530018070 con 0x7f15380c8190 2024-04-25T11:20:48.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530004d70 con 0x7f15380c8190 2024-04-25T11:20:48.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1538133800 con 0x7f15380c8190 2024-04-25T11:20:48.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153000a880 con 0x7f15380c8190 2024-04-25T11:20:48.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:48.996+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153815b0a0 con 0x7f15380c8190 2024-04-25T11:20:48.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153806a350 con 0x7f15380c8190 2024-04-25T11:20:48.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f153001b070 con 0x7f15380c8190 2024-04-25T11:20:48.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f151f7fe640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 0x7f1524071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:48.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f153008edc0 con 0x7f15380c8190 2024-04-25T11:20:48.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f1536d9d640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 0x7f1524071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:48.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.000+0000 7f1536d9d640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 0x7f1524071910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f152800b930 tx=0x7f1528008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:49.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.004+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f153005b960 con 0x7f15380c8190 2024-04-25T11:20:49.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.200+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f15380ccdc0 con 0x7f15380c8190 2024-04-25T11:20:49.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:20:49.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:20:49.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.832+0000 7f151f7fe640 1 -- 172.21.15.136:0/1578713336 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f153005b260 con 0x7f15380c8190 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 msgr2=0x7f1524071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 0x7f1524071910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f152800b930 tx=0x7f1528008040 comp rx=0 tx=0).stop 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 msgr2=0x7f1538153320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f153000c6d0 tx=0x7f153000cb90 comp rx=0 tx=0).stop 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 shutdown_connections 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380c8190 0x7f1538153320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f152406f400 0x7f1524071910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1538072dd0 0x7f1538152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:49.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 --2- 172.21.15.136:0/1578713336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15380729c0 0x7f1538139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:49.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 >> 172.21.15.136:0/1578713336 conn(0x7f15380c9d50 msgr2=0x7f15380ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:49.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 shutdown_connections 2024-04-25T11:20:49.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:49.836+0000 7f153d064640 1 -- 172.21.15.136:0/1578713336 wait complete. 2024-04-25T11:20:49.859 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-04-25T11:20:49.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T11:20:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 Processor -- start 2024-04-25T11:20:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 -- start start 2024-04-25T11:20:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65141562a0 0x7f65140ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65140cead0 0x7f65141669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651415c230 con 0x7f6514073b80 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f65141605e0 con 0x7f65141562a0 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f651a5fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65140c9320 con 0x7f65140cead0 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f65137fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65141562a0 0x7f65140ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f65137fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65141562a0 0x7f65140ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36868/0 (socket says 172.21.15.136:36868) 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f65137fe640 1 -- 172.21.15.136:0/2845664101 learned_addr learned my addr 172.21.15.136:0/2845664101 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.016+0000 7f6518b98640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65140cead0 0x7f65141669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f65137fe640 1 -- 172.21.15.136:0/2845664101 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65140cead0 msgr2=0x7f65141669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f65137fe640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65140cead0 0x7f65141669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f65137fe640 1 -- 172.21.15.136:0/2845664101 >> v1:172.21.15.157:6789/0 conn(0x7f6514073b80 legacy=0x7f651415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f65137fe640 1 -- 172.21.15.136:0/2845664101 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6514160020 con 0x7f65141562a0 2024-04-25T11:20:50.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f65137fe640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65141562a0 0x7f65140ce590 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37053e4f96676d37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:50.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f6512ffd640 1 -- 172.21.15.136:0/2845664101 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080b0070 con 0x7f65141562a0 2024-04-25T11:20:50.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f6512ffd640 1 -- 172.21.15.136:0/2845664101 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080ab600 con 0x7f65141562a0 2024-04-25T11:20:50.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f6512ffd640 1 -- 172.21.15.136:0/2845664101 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080ab9d0 con 0x7f65141562a0 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 -- 172.21.15.136:0/2845664101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65141562a0 msgr2=0x7f65140ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65141562a0 0x7f65140ce590 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c520 comp rx=0 tx=0).stop 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 -- 172.21.15.136:0/2845664101 shutdown_connections 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65140cead0 0x7f65141669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 --2- 172.21.15.136:0/2845664101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65141562a0 0x7f65140ce590 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 -- 172.21.15.136:0/2845664101 >> 172.21.15.136:0/2845664101 conn(0x7f651406d950 msgr2=0x7f651406dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 -- 172.21.15.136:0/2845664101 shutdown_connections 2024-04-25T11:20:50.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.020+0000 7f651a5fb640 1 -- 172.21.15.136:0/2845664101 wait complete. 2024-04-25T11:20:50.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 Processor -- start 2024-04-25T11:20:50.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- start start 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6514073b80 0x7f65141f8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65141562a0 0x7f651414f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f65140c8f70 con 0x7f65140cead0 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65141566b0 con 0x7f6514073b80 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65141f9690 con 0x7f65141562a0 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36872/0 (socket says 172.21.15.136:36872) 2024-04-25T11:20:50.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 -- 172.21.15.136:0/3212959102 learned_addr learned my addr 172.21.15.136:0/3212959102 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f6518b98640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65141562a0 0x7f651414f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f6513fff640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6514073b80 0x7f65141f8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 -- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65141562a0 msgr2=0x7f651414f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65141562a0 0x7f651414f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 -- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6514073b80 msgr2=0x7f65141f8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6514073b80 0x7f65141f8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 -- 172.21.15.136:0/3212959102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65141f91c0 con 0x7f65140cead0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f65137fe640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65080ae040 tx=0x7f65080192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080abe60 con 0x7f65140cead0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080b4640 con 0x7f65140cead0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6514156a30 con 0x7f65140cead0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080b0040 con 0x7f65140cead0 2024-04-25T11:20:50.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.024+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65141f93d0 con 0x7f65140cead0 2024-04-25T11:20:50.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651406a2e0 con 0x7f65140cead0 2024-04-25T11:20:50.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f65080ab4c0 con 0x7f65140cead0 2024-04-25T11:20:50.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f6510ff9640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 0x7f6500071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:50.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6508129090 con 0x7f65140cead0 2024-04-25T11:20:50.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f6513fff640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 0x7f6500071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:50.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.028+0000 7f6513fff640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 0x7f6500071910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f650c007bf0 tx=0x7f650c007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:50.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.032+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6508016110 con 0x7f65140cead0 2024-04-25T11:20:50.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.228+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f65140740f0 con 0x7f65140cead0 2024-04-25T11:20:50.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.848+0000 7f6510ff9640 1 -- 172.21.15.136:0/3212959102 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) ==== 105+0+0 (secure 0 0 0) 0x7f65080f5d60 con 0x7f65140cead0 2024-04-25T11:20:50.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 msgr2=0x7f6500071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 0x7f6500071910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f650c007bf0 tx=0x7f650c007240 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 msgr2=0x7f651415b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f65080ae040 tx=0x7f65080192c0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 shutdown_connections 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f650006f400 0x7f6500071910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65141562a0 0x7f651414f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65140cead0 0x7f651415b0b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 --2- 172.21.15.136:0/3212959102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6514073b80 0x7f65141f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 >> 172.21.15.136:0/3212959102 conn(0x7f651406d950 msgr2=0x7f651406dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 shutdown_connections 2024-04-25T11:20:50.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:50.852+0000 7f651a5fb640 1 -- 172.21.15.136:0/3212959102 wait complete. 2024-04-25T11:20:50.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T11:20:51.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.032+0000 7f7b1dd66640 1 Processor -- start 2024-04-25T11:20:51.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.032+0000 7f7b1dd66640 1 -- start start 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b1dd66640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b180c7c50 0x7f7b180c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b1dd66640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b181615c0 0x7f7b181639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b1dd66640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b18159900 con 0x7f7b180c8e40 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b1dd66640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7b181596d0 con 0x7f7b180c7c50 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b1dd66640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b18159bc0 con 0x7f7b181615c0 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b180c7c50 0x7f7b180c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b180c7c50 0x7f7b180c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36874/0 (socket says 172.21.15.136:36874) 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 -- 172.21.15.136:0/1818548802 learned_addr learned my addr 172.21.15.136:0/1818548802 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:51.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b17fff640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b181615c0 0x7f7b181639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 -- 172.21.15.136:0/1818548802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b181615c0 msgr2=0x7f7b181639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b181615c0 0x7f7b181639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 -- 172.21.15.136:0/1818548802 >> v1:172.21.15.157:6789/0 conn(0x7f7b180c8e40 legacy=0x7f7b180c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 -- 172.21.15.136:0/1818548802 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1815c3e0 con 0x7f7b180c7c50 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b177fe640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b180c7c50 0x7f7b180c8900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009870 tx=0x7f7b0801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd54edd13063ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:51.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b167fc640 1 -- 172.21.15.136:0/1818548802 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b080b5070 con 0x7f7b180c7c50 2024-04-25T11:20:51.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b167fc640 1 -- 172.21.15.136:0/1818548802 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b080a89d0 con 0x7f7b180c7c50 2024-04-25T11:20:51.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.036+0000 7f7b167fc640 1 -- 172.21.15.136:0/1818548802 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b080a8da0 con 0x7f7b180c7c50 2024-04-25T11:20:51.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- 172.21.15.136:0/1818548802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b180c7c50 msgr2=0x7f7b180c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:51.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b180c7c50 0x7f7b180c8900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009870 tx=0x7f7b0801ed10 comp rx=0 tx=0).stop 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- 172.21.15.136:0/1818548802 shutdown_connections 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b181615c0 0x7f7b181639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/1818548802 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b180c7c50 0x7f7b180c8900 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- 172.21.15.136:0/1818548802 >> 172.21.15.136:0/1818548802 conn(0x7f7b180c9d50 msgr2=0x7f7b180ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- 172.21.15.136:0/1818548802 shutdown_connections 2024-04-25T11:20:51.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- 172.21.15.136:0/1818548802 wait complete. 2024-04-25T11:20:51.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 Processor -- start 2024-04-25T11:20:51.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- start start 2024-04-25T11:20:51.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b180c7c50 0x7f7b181620d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b180c8e40 0x7f7b180cc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1815cc80 con 0x7f7b181615c0 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1815c9a0 con 0x7f7b180c8e40 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b1dd66640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b180c5270 con 0x7f7b180c7c50 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36882/0 (socket says 172.21.15.136:36882) 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 -- 172.21.15.136:0/4266958444 learned_addr learned my addr 172.21.15.136:0/4266958444 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b177fe640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b180c7c50 0x7f7b181620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b16ffd640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b180c8e40 0x7f7b180cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 -- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b180c7c50 msgr2=0x7f7b181620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b180c7c50 0x7f7b181620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 -- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b180c8e40 msgr2=0x7f7b180cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:51.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b180c8e40 0x7f7b180cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.040+0000 7f7b17fff640 1 -- 172.21.15.136:0/4266958444 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b18162610 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b17fff640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007d70 tx=0x7f7b1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10014420 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b10019070 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b18150a70 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b10014820 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b18151030 con 0x7f7b181615c0 2024-04-25T11:20:51.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b181627e0 con 0x7f7b181615c0 2024-04-25T11:20:51.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7b100149c0 con 0x7f7b181615c0 2024-04-25T11:20:51.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 0x7f7b04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:51.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7b1008fdc0 con 0x7f7b181615c0 2024-04-25T11:20:51.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b177fe640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 0x7f7b04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:51.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.044+0000 7f7b177fe640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 0x7f7b04071910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b2000 tx=0x7f7b08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:51.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.048+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7b1005ca00 con 0x7f7b181615c0 2024-04-25T11:20:51.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.232+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7b1806a350 con 0x7f7b181615c0 2024-04-25T11:20:51.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.860+0000 7f7afffff640 1 -- 172.21.15.136:0/4266958444 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f7b1005c300 con 0x7f7b181615c0 2024-04-25T11:20:51.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 msgr2=0x7f7b04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:51.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 0x7f7b04071910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7b080b2000 tx=0x7f7b08008040 comp rx=0 tx=0).stop 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 msgr2=0x7f7b1815b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b10007d70 tx=0x7f7b1000e4b0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 shutdown_connections 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b181615c0 0x7f7b1815b070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b180c8e40 0x7f7b180cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0406f400 0x7f7b04071910 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 --2- 172.21.15.136:0/4266958444 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b180c7c50 0x7f7b181620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:51.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 >> 172.21.15.136:0/4266958444 conn(0x7f7b180c9d50 msgr2=0x7f7b18157180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:51.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 shutdown_connections 2024-04-25T11:20:51.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:51.864+0000 7f7b1dd66640 1 -- 172.21.15.136:0/4266958444 wait complete. 2024-04-25T11:20:51.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T11:20:52.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 Processor -- start 2024-04-25T11:20:52.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 -- start start 2024-04-25T11:20:52.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b940c96e0 0x7f4b941540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4b9415aae0 0x7f4b940c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b940c9af0 con 0x7f4b940c79e0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f4b9406cc10 con 0x7f4b940c96e0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b9a9ba640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4b940cb220 con 0x7f4b9415aae0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b93fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b940c96e0 0x7f4b941540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b937fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4b940c79e0 0x7f4b9415a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53276/0 (socket says 172.21.15.136:53276) 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b937fe640 1 -- 172.21.15.136:0/303780375 learned_addr learned my addr 172.21.15.136:0/303780375 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b98f57640 1 --2- 172.21.15.136:0/303780375 >> v2:172.21.15.157:3300/0 conn(0x7f4b9415aae0 0x7f4b940c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b92ffd640 1 -- 172.21.15.136:0/303780375 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480715490 0 0) 0x7f4b940c9af0 con 0x7f4b940c79e0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b92ffd640 1 -- 172.21.15.136:0/303780375 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b7c003530 con 0x7f4b940c79e0 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b93fff640 1 -- 172.21.15.136:0/303780375 >> v2:172.21.15.157:3300/0 conn(0x7f4b9415aae0 msgr2=0x7f4b940c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b93fff640 1 --2- 172.21.15.136:0/303780375 >> v2:172.21.15.157:3300/0 conn(0x7f4b9415aae0 0x7f4b940c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b93fff640 1 -- 172.21.15.136:0/303780375 >> v1:172.21.15.136:6789/0 conn(0x7f4b940c79e0 legacy=0x7f4b9415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.052+0000 7f4b93fff640 1 -- 172.21.15.136:0/303780375 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b940cb500 con 0x7f4b940c96e0 2024-04-25T11:20:52.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b93fff640 1 --2- 172.21.15.136:0/303780375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c96e0 0x7f4b941540d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4b94075140 tx=0x7f4b8401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99223599c553b523 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:52.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b92ffd640 1 -- 172.21.15.136:0/303780375 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b5070 con 0x7f4b940c96e0 2024-04-25T11:20:52.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b92ffd640 1 -- 172.21.15.136:0/303780375 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840a8970 con 0x7f4b940c96e0 2024-04-25T11:20:52.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b92ffd640 1 -- 172.21.15.136:0/303780375 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840a8d40 con 0x7f4b940c96e0 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/303780375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c96e0 msgr2=0x7f4b941540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/303780375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c96e0 0x7f4b941540d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4b94075140 tx=0x7f4b8401ec70 comp rx=0 tx=0).stop 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/303780375 shutdown_connections 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/303780375 >> v2:172.21.15.157:3300/0 conn(0x7f4b9415aae0 0x7f4b940c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/303780375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c96e0 0x7f4b941540d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/303780375 >> 172.21.15.136:0/303780375 conn(0x7f4b9406f9d0 msgr2=0x7f4b9406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/303780375 shutdown_connections 2024-04-25T11:20:52.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.056+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/303780375 wait complete. 2024-04-25T11:20:52.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 Processor -- start 2024-04-25T11:20:52.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- start start 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 0x7f4b941f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b9415aae0 0x7f4b94140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4b940cf0f0 con 0x7f4b940c79e0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b94154610 con 0x7f4b940c96e0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4b940ced40 con 0x7f4b9415aae0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36896/0 (socket says 172.21.15.136:36896) 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 -- 172.21.15.136:0/2948060608 learned_addr learned my addr 172.21.15.136:0/2948060608 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b937fe640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 0x7f4b941f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b98f57640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b9415aae0 0x7f4b94140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 -- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b9415aae0 msgr2=0x7f4b94140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b9415aae0 0x7f4b94140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 -- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 msgr2=0x7f4b941f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 0x7f4b941f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 -- 172.21.15.136:0/2948060608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b941f4260 con 0x7f4b940c79e0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b937fe640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 0x7f4b941f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b93fff640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b8401c8a0 tx=0x7f4b8401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840bf070 con 0x7f4b940c79e0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840b0720 con 0x7f4b940c79e0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b94159b20 con 0x7f4b940c79e0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b5040 con 0x7f4b940c79e0 2024-04-25T11:20:52.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9414ece0 con 0x7f4b940c79e0 2024-04-25T11:20:52.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.060+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9406a350 con 0x7f4b940c79e0 2024-04-25T11:20:52.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.064+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4b840c2070 con 0x7f4b940c79e0 2024-04-25T11:20:52.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.064+0000 7f4b90ff9640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 0x7f4b80071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:52.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.064+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4b8412bc90 con 0x7f4b940c79e0 2024-04-25T11:20:52.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.064+0000 7f4b937fe640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 0x7f4b80071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:52.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.064+0000 7f4b937fe640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 0x7f4b80071910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4b94158d70 tx=0x7f4b88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:52.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.068+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4b840187d0 con 0x7f4b940c79e0 2024-04-25T11:20:52.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.264+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f4b940c55b0 con 0x7f4b940c79e0 2024-04-25T11:20:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.872+0000 7f4b90ff9640 1 -- 172.21.15.136:0/2948060608 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) ==== 115+0+0 (secure 0 0 0) 0x7f4b840f8840 con 0x7f4b940c79e0 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 msgr2=0x7f4b80071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 0x7f4b80071910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4b94158d70 tx=0x7f4b88009040 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 msgr2=0x7f4b94143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b8401c8a0 tx=0x7f4b8401ba10 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 shutdown_connections 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b8006f400 0x7f4b80071910 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b9415aae0 0x7f4b94140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b940c96e0 0x7f4b941f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 --2- 172.21.15.136:0/2948060608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b940c79e0 0x7f4b94143af0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 >> 172.21.15.136:0/2948060608 conn(0x7f4b9406f9d0 msgr2=0x7f4b9406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:52.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 shutdown_connections 2024-04-25T11:20:52.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:52.876+0000 7f4b9a9ba640 1 -- 172.21.15.136:0/2948060608 wait complete. 2024-04-25T11:20:52.899 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T11:20:52.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:20:53.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 Processor -- start 2024-04-25T11:20:53.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 -- start start 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f98b00c8cd0 0x7f98b014f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00ca720 0x7f98b00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b00cbad0 con 0x7f98b00ce260 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f98b00c8550 con 0x7f98b00c8cd0 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b84b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f98b00c7c20 con 0x7f98b00ca720 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b5a54640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f98b00ce260 0x7f98b00ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53282/0 (socket says 172.21.15.136:53282) 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b5a54640 1 -- 172.21.15.136:0/786214023 learned_addr learned my addr 172.21.15.136:0/786214023 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6255640 1 --2- 172.21.15.136:0/786214023 >> v2:172.21.15.136:3300/0 conn(0x7f98b00c8cd0 0x7f98b014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6a56640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00ca720 0x7f98b00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b5253640 1 -- 172.21.15.136:0/786214023 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240957101 0 0) 0x7f98b00cbad0 con 0x7f98b00ce260 2024-04-25T11:20:53.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b5253640 1 -- 172.21.15.136:0/786214023 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9898003530 con 0x7f98b00ce260 2024-04-25T11:20:53.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6255640 1 -- 172.21.15.136:0/786214023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00ca720 msgr2=0x7f98b00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6255640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00ca720 0x7f98b00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6255640 1 -- 172.21.15.136:0/786214023 >> v1:172.21.15.136:6789/0 conn(0x7f98b00ce260 legacy=0x7f98b00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.052+0000 7f98b6255640 1 -- 172.21.15.136:0/786214023 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b00c7f00 con 0x7f98b00c8cd0 2024-04-25T11:20:53.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b6255640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00c8cd0 0x7f98b014f680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c86d0 tx=0x7f98a801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597ba8dd25298aea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b5253640 1 -- 172.21.15.136:0/786214023 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80b5070 con 0x7f98b00c8cd0 2024-04-25T11:20:53.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b5253640 1 -- 172.21.15.136:0/786214023 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a80a8990 con 0x7f98b00c8cd0 2024-04-25T11:20:53.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b5253640 1 -- 172.21.15.136:0/786214023 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80a8d60 con 0x7f98b00c8cd0 2024-04-25T11:20:53.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 -- 172.21.15.136:0/786214023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00c8cd0 msgr2=0x7f98b014f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00c8cd0 0x7f98b014f680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c86d0 tx=0x7f98a801ec90 comp rx=0 tx=0).stop 2024-04-25T11:20:53.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 -- 172.21.15.136:0/786214023 shutdown_connections 2024-04-25T11:20:53.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00ca720 0x7f98b00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 --2- 172.21.15.136:0/786214023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00c8cd0 0x7f98b014f680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 -- 172.21.15.136:0/786214023 >> 172.21.15.136:0/786214023 conn(0x7f98b006f9d0 msgr2=0x7f98b006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 -- 172.21.15.136:0/786214023 shutdown_connections 2024-04-25T11:20:53.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.056+0000 7f98b84b9640 1 -- 172.21.15.136:0/786214023 wait complete. 2024-04-25T11:20:53.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 Processor -- start 2024-04-25T11:20:53.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- start start 2024-04-25T11:20:53.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 0x7f98b0156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98b00ce260 0x7f98b0144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f98b00cb720 con 0x7f98b00ca720 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f98b014fbc0 con 0x7f98b00ce260 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f98b01e2bc0 con 0x7f98b00c8cd0 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36910/0 (socket says 172.21.15.136:36910) 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 -- 172.21.15.136:0/778936066 learned_addr learned my addr 172.21.15.136:0/778936066 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b6255640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 0x7f98b0156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 -- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 msgr2=0x7f98b0156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 0x7f98b0156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b6a56640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98b00ce260 0x7f98b0144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 -- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98b00ce260 msgr2=0x7f98b0144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98b00ce260 0x7f98b0144620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 -- 172.21.15.136:0/778936066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b01e2e80 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b6255640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 0x7f98b0156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b5a54640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98a400caa0 tx=0x7f98a400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4017070 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a400aba0 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b01ef170 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4012400 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b01ef730 con 0x7f98b00ca720 2024-04-25T11:20:53.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.060+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b01eef60 con 0x7f98b00ca720 2024-04-25T11:20:53.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.064+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f98a401a070 con 0x7f98b00ca720 2024-04-25T11:20:53.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.064+0000 7f98aeffd640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 0x7f989c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.064+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f98a408f270 con 0x7f98b00ca720 2024-04-25T11:20:53.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.064+0000 7f98b6255640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 0x7f989c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.064+0000 7f98b6255640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 0x7f989c071940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98a80b2000 tx=0x7f98a80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.068+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f98a405bf40 con 0x7f98b00ca720 2024-04-25T11:20:53.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.260+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98b006a350 con 0x7f98b00ca720 2024-04-25T11:20:53.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.260+0000 7f98aeffd640 1 -- 172.21.15.136:0/778936066 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6854 (secure 0 0 0) 0x7f98a405b840 con 0x7f98b00ca720 2024-04-25T11:20:53.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:53.260 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:20:52.268990+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4303,"mds_1":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:creating","state_seq":1,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303]},"id":1}]} 2024-04-25T11:20:53.260 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 12 2024-04-25T11:20:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 msgr2=0x7f989c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 0x7f989c071940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98a80b2000 tx=0x7f98a80aa070 comp rx=0 tx=0).stop 2024-04-25T11:20:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 msgr2=0x7f98b0157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98a400caa0 tx=0x7f98a400cf60 comp rx=0 tx=0).stop 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 shutdown_connections 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98b00ce260 0x7f98b0144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98b00ca720 0x7f98b0157220 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f989c06f430 0x7f989c071940 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 --2- 172.21.15.136:0/778936066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f98b00c8cd0 0x7f98b0156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 >> 172.21.15.136:0/778936066 conn(0x7f98b006f9d0 msgr2=0x7f98b015c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 shutdown_connections 2024-04-25T11:20:53.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.264+0000 7f98b84b9640 1 -- 172.21.15.136:0/778936066 wait complete. 2024-04-25T11:20:53.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:53.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 Processor -- start 2024-04-25T11:20:53.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 -- start start 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fef140689a0 0x7fef14153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef140b3720 con 0x7fef14074970 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef140b3a00 con 0x7fef14068590 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef1a231640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fef14073b30 con 0x7fef140689a0 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef137fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fef14074970 0x7fef14159f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53298/0 (socket says 172.21.15.136:53298) 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef137fe640 1 -- 172.21.15.136:0/2260142620 learned_addr learned my addr 172.21.15.136:0/2260142620 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef13fff640 1 --2- 172.21.15.136:0/2260142620 >> v2:172.21.15.157:3300/0 conn(0x7fef140689a0 0x7fef14153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3923141384 0 0) 0x7fef140b3720 con 0x7fef14074970 2024-04-25T11:20:53.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.428+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feefc003530 con 0x7fef14074970 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 88637554 0 0) 0x7feefc003530 con 0x7fef14074970 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 >> v2:172.21.15.157:3300/0 conn(0x7fef140689a0 msgr2=0x7fef14153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 --2- 172.21.15.136:0/2260142620 >> v2:172.21.15.157:3300/0 conn(0x7fef140689a0 0x7fef14153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 >> v1:172.21.15.157:6789/0 conn(0x7fef14068590 legacy=0x7fef14073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef140b3ce0 con 0x7fef14074970 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fef08001680 con 0x7fef14074970 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 >> v1:172.21.15.136:6789/0 conn(0x7fef14074970 legacy=0x7fef14159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feefc0039c0 0x7feefc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7feefc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feefc005280 0x7feefc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fef14073b30 con 0x7feefc004810 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feefc00c1a0 con 0x7feefc005280 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feefc00c3a0 con 0x7feefc0039c0 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7feefc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef137fe640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feefc0039c0 0x7feefc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef13fff640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feefc005280 0x7feefc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 -- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feefc0039c0 msgr2=0x7feefc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feefc0039c0 0x7feefc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 -- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feefc005280 msgr2=0x7feefc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feefc005280 0x7feefc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef13fff640 1 -- 172.21.15.136:0/2260142620 reap_dead start 2024-04-25T11:20:53.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 -- 172.21.15.136:0/2260142620 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feefc003530 con 0x7feefc004810 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef12ffd640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7feefc004c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef1406f070 tx=0x7fef0401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5286d846dba834c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040013a0 con 0x7feefc004810 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef040052f0 con 0x7feefc004810 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef127fc640 1 -- 172.21.15.136:0/2260142620 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040a4e60 con 0x7feefc004810 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 -- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 msgr2=0x7feefc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7feefc004c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef1406f070 tx=0x7fef0401c3a0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 -- 172.21.15.136:0/2260142620 shutdown_connections 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 --2- 172.21.15.136:0/2260142620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7feefc004c20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 -- 172.21.15.136:0/2260142620 >> 172.21.15.136:0/2260142620 conn(0x7fef140ca640 msgr2=0x7fef140caa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 -- 172.21.15.136:0/2260142620 shutdown_connections 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.432+0000 7fef1a231640 1 -- 172.21.15.136:0/2260142620 wait complete. 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 Processor -- start 2024-04-25T11:20:53.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- start start 2024-04-25T11:20:53.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef14147c80 0x7fef1414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fef141470f0 0x7fef14147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fef14074450 con 0x7feefc004810 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fef14074170 con 0x7fef14147c80 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fef140cb440 con 0x7fef141470f0 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36928/0 (socket says 172.21.15.136:36928) 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 -- 172.21.15.136:0/76822641 learned_addr learned my addr 172.21.15.136:0/76822641 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef12ffd640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef14147c80 0x7fef1414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef13fff640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fef141470f0 0x7fef14147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 -- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fef141470f0 msgr2=0x7fef14147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fef141470f0 0x7fef14147500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 -- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef14147c80 msgr2=0x7fef1414e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef14147c80 0x7fef1414e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 -- 172.21.15.136:0/76822641 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1414f1f0 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef137fe640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef080119b0 tx=0x7fef08011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08010040 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef0801a070 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1414c520 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0800fe20 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.436+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1414c7e0 con 0x7feefc004810 2024-04-25T11:20:53.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef1406a350 con 0x7feefc004810 2024-04-25T11:20:53.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fef0801f460 con 0x7feefc004810 2024-04-25T11:20:53.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7feefbfff640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 0x7fef00071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fef08092890 con 0x7feefc004810 2024-04-25T11:20:53.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7fef12ffd640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 0x7fef00071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.440+0000 7fef12ffd640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 0x7fef00071910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef1406fdc0 tx=0x7fef040011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.444+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fef0805f590 con 0x7feefc004810 2024-04-25T11:20:53.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.604+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef14147a40 con 0x7feefc004810 2024-04-25T11:20:53.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.608+0000 7feefbfff640 1 -- 172.21.15.136:0/76822641 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fef0805ee90 con 0x7feefc004810 2024-04-25T11:20:53.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:53.607 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 msgr2=0x7fef00071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 0x7fef00071910 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef1406fdc0 tx=0x7fef040011a0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 msgr2=0x7fef141f3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef080119b0 tx=0x7fef08011e70 comp rx=0 tx=0).stop 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 shutdown_connections 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fef0006f400 0x7fef00071910 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fef141470f0 0x7fef14147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fef14147c80 0x7fef1414e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 --2- 172.21.15.136:0/76822641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feefc004810 0x7fef141f3e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 >> 172.21.15.136:0/76822641 conn(0x7fef140ca640 msgr2=0x7fef14151d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 shutdown_connections 2024-04-25T11:20:53.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.612+0000 7fef1a231640 1 -- 172.21.15.136:0/76822641 wait complete. 2024-04-25T11:20:53.629 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T11:20:53.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:20:53.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.768+0000 7ff5d4a94640 1 Processor -- start 2024-04-25T11:20:53.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.768+0000 7ff5d4a94640 1 -- start start 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5d4a94640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5d015d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5d4a94640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff5d0070c80 0x7ff5d0164790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5d4a94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d0143580 con 0x7ff5d0070870 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5d4a94640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d015dd20 con 0x7ff5d0070c80 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5d4a94640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d015e000 con 0x7ff5d0143170 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cdd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff5d0070870 0x7ff5d00c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53314/0 (socket says 172.21.15.136:53314) 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cdd9b640 1 -- 172.21.15.136:0/2411924100 learned_addr learned my addr 172.21.15.136:0/2411924100 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5d015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ced9d640 1 --2- 172.21.15.136:0/2411924100 >> v2:172.21.15.157:3300/0 conn(0x7ff5d0070c80 0x7ff5d0164790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1335121777 0 0) 0x7ff5c4001120 con 0x7ff5d0070870 2024-04-25T11:20:53.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5b8003530 con 0x7ff5d0070870 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3116660664 0 0) 0x7ff5b8003530 con 0x7ff5d0070870 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 msgr2=0x7ff5d015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5d015d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 >> v2:172.21.15.157:3300/0 conn(0x7ff5d0070c80 msgr2=0x7ff5d0164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 --2- 172.21.15.136:0/2411924100 >> v2:172.21.15.157:3300/0 conn(0x7ff5d0070c80 0x7ff5d0164790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d006fd90 con 0x7ff5d0070870 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7ff5c4001850 con 0x7ff5d0070870 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 >> v1:172.21.15.136:6789/0 conn(0x7ff5d0070870 legacy=0x7ff5d00c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5b8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5b8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5b8004d50 0x7ff5b8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d015dd20 con 0x7ff5b80039c0 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d015e000 con 0x7ff5b8004d50 2024-04-25T11:20:53.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b800ba50 con 0x7ff5d0143170 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5b8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5cdd9b640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5b8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ced9d640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5b8004d50 0x7ff5b8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 -- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 msgr2=0x7ff5b8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0143170 0x7ff5b8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 -- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5b8004d50 msgr2=0x7ff5b8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5b8004d50 0x7ff5b8005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.772+0000 7ff5ce59c640 1 -- 172.21.15.136:0/2411924100 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b8003530 con 0x7ff5b80039c0 2024-04-25T11:20:53.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5b8003e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009ab0 tx=0x7ff5c001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1f6b6a090e55e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c00b2070 con 0x7ff5b80039c0 2024-04-25T11:20:53.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c00a5660 con 0x7ff5b80039c0 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5cd59a640 1 -- 172.21.15.136:0/2411924100 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c00096b0 con 0x7ff5b80039c0 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- 172.21.15.136:0/2411924100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 msgr2=0x7ff5b8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/2411924100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5b8003e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009ab0 tx=0x7ff5c001ca70 comp rx=0 tx=0).stop 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5ced9d640 1 -- 172.21.15.136:0/2411924100 reap_dead start 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- 172.21.15.136:0/2411924100 shutdown_connections 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- 172.21.15.136:0/2411924100 >> 172.21.15.136:0/2411924100 conn(0x7ff5d00c9d50 msgr2=0x7ff5d00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- 172.21.15.136:0/2411924100 shutdown_connections 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- 172.21.15.136:0/2411924100 wait complete. 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 Processor -- start 2024-04-25T11:20:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.776+0000 7ff5d4a94640 1 -- start start 2024-04-25T11:20:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0142700 0x7ff5d0140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5d0158440 0x7ff5d0158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d015e2e0 con 0x7ff5b80039c0 2024-04-25T11:20:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0070350 con 0x7ff5d0158440 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d00712a0 con 0x7ff5d0142700 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36944/0 (socket says 172.21.15.136:36944) 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 -- 172.21.15.136:0/288648263 learned_addr learned my addr 172.21.15.136:0/288648263 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5cdd9b640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0142700 0x7ff5d0140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ced9d640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5d0158440 0x7ff5d0158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 -- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0142700 msgr2=0x7ff5d0140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0142700 0x7ff5d0140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 -- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5d0158440 msgr2=0x7ff5d0158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5d0158440 0x7ff5d0158850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 -- 172.21.15.136:0/288648263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d01f0b30 con 0x7ff5b80039c0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5ce59c640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00b0040 tx=0x7ff5b8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c00b2040 con 0x7ff5b80039c0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c00a5e30 con 0x7ff5b80039c0 2024-04-25T11:20:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0004020 con 0x7ff5b80039c0 2024-04-25T11:20:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d0141b30 con 0x7ff5b80039c0 2024-04-25T11:20:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d01420f0 con 0x7ff5b80039c0 2024-04-25T11:20:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d006a350 con 0x7ff5b80039c0 2024-04-25T11:20:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff5c00c0030 con 0x7ff5b80039c0 2024-04-25T11:20:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 0x7ff5b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:53.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.780+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff5c0129bb0 con 0x7ff5b80039c0 2024-04-25T11:20:53.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.784+0000 7ff5cdd9b640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 0x7ff5b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:53.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.784+0000 7ff5cdd9b640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 0x7ff5b0071910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0142c90 tx=0x7ff5c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:53.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.784+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff5c00c7060 con 0x7ff5b80039c0 2024-04-25T11:20:53.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.932+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5d0074340 con 0x7ff5b006f400 2024-04-25T11:20:53.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.976+0000 7ff5beffd640 1 -- 172.21.15.136:0/288648263 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332527 (secure 0 0 0) 0x7ff5d0074340 con 0x7ff5b006f400 2024-04-25T11:20:53.977 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:53.980 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 msgr2=0x7ff5b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 0x7ff5b0071910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0142c90 tx=0x7ff5c4009040 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 msgr2=0x7ff5d01f05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff5c00b0040 tx=0x7ff5b8005840 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 shutdown_connections 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5d0158440 0x7ff5d0158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff5b006f400 0x7ff5b0071910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff5d0142700 0x7ff5d0140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 --2- 172.21.15.136:0/288648263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5b80039c0 0x7ff5d01f05f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 >> 172.21.15.136:0/288648263 conn(0x7ff5d00c9d50 msgr2=0x7ff5d00c8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 shutdown_connections 2024-04-25T11:20:53.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:53.980+0000 7ff5d4a94640 1 -- 172.21.15.136:0/288648263 wait complete. 2024-04-25T11:20:53.996 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-25T11:20:52.765071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":150,"num_write_kb":1647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236000,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738720,"statfs":{"total":773094113280,"available":772852449280,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005363"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070611+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:25.070611+0000","last_peered":"2024-04-25T11:20:25.070611+0000","last_clean":"2024-04-25T11:20:25.070611+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:25.070611+0000","last_undegraded":"2024-04-25T11:20:25.070611+0000","last_fullsized":"2024-04-25T11:20:25.070611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066703+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:25.066703+0000","last_peered":"2024-04-25T11:20:25.066703+0000","last_clean":"2024-04-25T11:20:25.066703+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:25.066703+0000","last_undegraded":"2024-04-25T11:20:25.066703+0000","last_fullsized":"2024-04-25T11:20:25.066703+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070831+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:25.070831+0000","last_peered":"2024-04-25T11:20:25.070831+0000","last_clean":"2024-04-25T11:20:25.070831+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:25.070831+0000","last_undegraded":"2024-04-25T11:20:25.070831+0000","last_fullsized":"2024-04-25T11:20:25.070831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848527+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:25.848527+0000","last_peered":"2024-04-25T11:20:25.848527+0000","last_clean":"2024-04-25T11:20:25.848527+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:25.848527+0000","last_undegraded":"2024-04-25T11:20:25.848527+0000","last_fullsized":"2024-04-25T11:20:25.848527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850005+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:20:25.850005+0000","last_peered":"2024-04-25T11:20:25.850005+0000","last_clean":"2024-04-25T11:20:25.850005+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:20:25.850005+0000","last_undegraded":"2024-04-25T11:20:25.850005+0000","last_fullsized":"2024-04-25T11:20:25.850005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852853+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:20:25.852853+0000","last_peered":"2024-04-25T11:20:25.852853+0000","last_clean":"2024-04-25T11:20:25.852853+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.852853+0000","last_undegraded":"2024-04-25T11:20:25.852853+0000","last_fullsized":"2024-04-25T11:20:25.852853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703772+0000","last_change":"2024-04-25T11:20:20.032982+0000","last_active":"2024-04-25T11:20:25.703772+0000","last_peered":"2024-04-25T11:20:25.703772+0000","last_clean":"2024-04-25T11:20:25.703772+0000","last_became_active":"2024-04-25T11:20:20.032127+0000","last_became_peered":"2024-04-25T11:20:20.032127+0000","last_unstale":"2024-04-25T11:20:25.703772+0000","last_undegraded":"2024-04-25T11:20:25.703772+0000","last_fullsized":"2024-04-25T11:20:25.703772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852476+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:20:25.852476+0000","last_peered":"2024-04-25T11:20:25.852476+0000","last_clean":"2024-04-25T11:20:25.852476+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:20:25.852476+0000","last_undegraded":"2024-04-25T11:20:25.852476+0000","last_fullsized":"2024-04-25T11:20:25.852476+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851378+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:20:25.851378+0000","last_peered":"2024-04-25T11:20:25.851378+0000","last_clean":"2024-04-25T11:20:25.851378+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:20:25.851378+0000","last_undegraded":"2024-04-25T11:20:25.851378+0000","last_fullsized":"2024-04-25T11:20:25.851378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704317+0000","last_change":"2024-04-25T11:20:20.031174+0000","last_active":"2024-04-25T11:20:25.704317+0000","last_peered":"2024-04-25T11:20:25.704317+0000","last_clean":"2024-04-25T11:20:25.704317+0000","last_became_active":"2024-04-25T11:20:20.030340+0000","last_became_peered":"2024-04-25T11:20:20.030340+0000","last_unstale":"2024-04-25T11:20:25.704317+0000","last_undegraded":"2024-04-25T11:20:25.704317+0000","last_fullsized":"2024-04-25T11:20:25.704317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850069+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:20:25.850069+0000","last_peered":"2024-04-25T11:20:25.850069+0000","last_clean":"2024-04-25T11:20:25.850069+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:20:25.850069+0000","last_undegraded":"2024-04-25T11:20:25.850069+0000","last_fullsized":"2024-04-25T11:20:25.850069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065389+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:25.065389+0000","last_peered":"2024-04-25T11:20:25.065389+0000","last_clean":"2024-04-25T11:20:25.065389+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:25.065389+0000","last_undegraded":"2024-04-25T11:20:25.065389+0000","last_fullsized":"2024-04-25T11:20:25.065389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850811+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:20:25.850811+0000","last_peered":"2024-04-25T11:20:25.850811+0000","last_clean":"2024-04-25T11:20:25.850811+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:20:25.850811+0000","last_undegraded":"2024-04-25T11:20:25.850811+0000","last_fullsized":"2024-04-25T11:20:25.850811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851666+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:20:25.851666+0000","last_peered":"2024-04-25T11:20:25.851666+0000","last_clean":"2024-04-25T11:20:25.851666+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:20:25.851666+0000","last_undegraded":"2024-04-25T11:20:25.851666+0000","last_fullsized":"2024-04-25T11:20:25.851666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067265+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:25.067265+0000","last_peered":"2024-04-25T11:20:25.067265+0000","last_clean":"2024-04-25T11:20:25.067265+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:25.067265+0000","last_undegraded":"2024-04-25T11:20:25.067265+0000","last_fullsized":"2024-04-25T11:20:25.067265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850338+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:20:25.850338+0000","last_peered":"2024-04-25T11:20:25.850338+0000","last_clean":"2024-04-25T11:20:25.850338+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:20:25.850338+0000","last_undegraded":"2024-04-25T11:20:25.850338+0000","last_fullsized":"2024-04-25T11:20:25.850338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850575+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:20:25.850575+0000","last_peered":"2024-04-25T11:20:25.850575+0000","last_clean":"2024-04-25T11:20:25.850575+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:20:25.850575+0000","last_undegraded":"2024-04-25T11:20:25.850575+0000","last_fullsized":"2024-04-25T11:20:25.850575+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066984+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:25.066984+0000","last_peered":"2024-04-25T11:20:25.066984+0000","last_clean":"2024-04-25T11:20:25.066984+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:25.066984+0000","last_undegraded":"2024-04-25T11:20:25.066984+0000","last_fullsized":"2024-04-25T11:20:25.066984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848318+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:20:25.848318+0000","last_peered":"2024-04-25T11:20:25.848318+0000","last_clean":"2024-04-25T11:20:25.848318+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:20:25.848318+0000","last_undegraded":"2024-04-25T11:20:25.848318+0000","last_fullsized":"2024-04-25T11:20:25.848318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850089+0000","last_change":"2024-04-25T11:20:24.165156+0000","last_active":"2024-04-25T11:20:25.850089+0000","last_peered":"2024-04-25T11:20:25.850089+0000","last_clean":"2024-04-25T11:20:25.850089+0000","last_became_active":"2024-04-25T11:20:24.164587+0000","last_became_peered":"2024-04-25T11:20:24.164587+0000","last_unstale":"2024-04-25T11:20:25.850089+0000","last_undegraded":"2024-04-25T11:20:25.850089+0000","last_fullsized":"2024-04-25T11:20:25.850089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851436+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:20:25.851436+0000","last_peered":"2024-04-25T11:20:25.851436+0000","last_clean":"2024-04-25T11:20:25.851436+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:20:25.851436+0000","last_undegraded":"2024-04-25T11:20:25.851436+0000","last_fullsized":"2024-04-25T11:20:25.851436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849318+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:20:25.849318+0000","last_peered":"2024-04-25T11:20:25.849318+0000","last_clean":"2024-04-25T11:20:25.849318+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:20:25.849318+0000","last_undegraded":"2024-04-25T11:20:25.849318+0000","last_fullsized":"2024-04-25T11:20:25.849318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850973+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:20:25.850973+0000","last_peered":"2024-04-25T11:20:25.850973+0000","last_clean":"2024-04-25T11:20:25.850973+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:20:25.850973+0000","last_undegraded":"2024-04-25T11:20:25.850973+0000","last_fullsized":"2024-04-25T11:20:25.850973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849979+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:25.849979+0000","last_peered":"2024-04-25T11:20:25.849979+0000","last_clean":"2024-04-25T11:20:25.849979+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:25.849979+0000","last_undegraded":"2024-04-25T11:20:25.849979+0000","last_fullsized":"2024-04-25T11:20:25.849979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066481+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:20:25.066481+0000","last_peered":"2024-04-25T11:20:25.066481+0000","last_clean":"2024-04-25T11:20:25.066481+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:20:25.066481+0000","last_undegraded":"2024-04-25T11:20:25.066481+0000","last_fullsized":"2024-04-25T11:20:25.066481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850564+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:20:25.850564+0000","last_peered":"2024-04-25T11:20:25.850564+0000","last_clean":"2024-04-25T11:20:25.850564+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:20:25.850564+0000","last_undegraded":"2024-04-25T11:20:25.850564+0000","last_fullsized":"2024-04-25T11:20:25.850564+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071084+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:25.071084+0000","last_peered":"2024-04-25T11:20:25.071084+0000","last_clean":"2024-04-25T11:20:25.071084+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:25.071084+0000","last_undegraded":"2024-04-25T11:20:25.071084+0000","last_fullsized":"2024-04-25T11:20:25.071084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850091+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:20:25.850091+0000","last_peered":"2024-04-25T11:20:25.850091+0000","last_clean":"2024-04-25T11:20:25.850091+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:20:25.850091+0000","last_undegraded":"2024-04-25T11:20:25.850091+0000","last_fullsized":"2024-04-25T11:20:25.850091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066330+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:25.066330+0000","last_peered":"2024-04-25T11:20:25.066330+0000","last_clean":"2024-04-25T11:20:25.066330+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:25.066330+0000","last_undegraded":"2024-04-25T11:20:25.066330+0000","last_fullsized":"2024-04-25T11:20:25.066330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850885+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:25.850885+0000","last_peered":"2024-04-25T11:20:25.850885+0000","last_clean":"2024-04-25T11:20:25.850885+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:25.850885+0000","last_undegraded":"2024-04-25T11:20:25.850885+0000","last_fullsized":"2024-04-25T11:20:25.850885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853257+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:20:25.853257+0000","last_peered":"2024-04-25T11:20:25.853257+0000","last_clean":"2024-04-25T11:20:25.853257+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.853257+0000","last_undegraded":"2024-04-25T11:20:25.853257+0000","last_fullsized":"2024-04-25T11:20:25.853257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848798+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:25.848798+0000","last_peered":"2024-04-25T11:20:25.848798+0000","last_clean":"2024-04-25T11:20:25.848798+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:25.848798+0000","last_undegraded":"2024-04-25T11:20:25.848798+0000","last_fullsized":"2024-04-25T11:20:25.848798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704541+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.704541+0000","last_peered":"2024-04-25T11:20:25.704541+0000","last_clean":"2024-04-25T11:20:25.704541+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:20:25.704541+0000","last_undegraded":"2024-04-25T11:20:25.704541+0000","last_fullsized":"2024-04-25T11:20:25.704541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850957+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:25.850957+0000","last_peered":"2024-04-25T11:20:25.850957+0000","last_clean":"2024-04-25T11:20:25.850957+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:25.850957+0000","last_undegraded":"2024-04-25T11:20:25.850957+0000","last_fullsized":"2024-04-25T11:20:25.850957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848973+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:25.848973+0000","last_peered":"2024-04-25T11:20:25.848973+0000","last_clean":"2024-04-25T11:20:25.848973+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:25.848973+0000","last_undegraded":"2024-04-25T11:20:25.848973+0000","last_fullsized":"2024-04-25T11:20:25.848973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852994+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:20:25.852994+0000","last_peered":"2024-04-25T11:20:25.852994+0000","last_clean":"2024-04-25T11:20:25.852994+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:20:25.852994+0000","last_undegraded":"2024-04-25T11:20:25.852994+0000","last_fullsized":"2024-04-25T11:20:25.852994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703782+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:20:25.703782+0000","last_peered":"2024-04-25T11:20:25.703782+0000","last_clean":"2024-04-25T11:20:25.703782+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:20:25.703782+0000","last_undegraded":"2024-04-25T11:20:25.703782+0000","last_fullsized":"2024-04-25T11:20:25.703782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850807+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:25.850807+0000","last_peered":"2024-04-25T11:20:25.850807+0000","last_clean":"2024-04-25T11:20:25.850807+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:25.850807+0000","last_undegraded":"2024-04-25T11:20:25.850807+0000","last_fullsized":"2024-04-25T11:20:25.850807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850780+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:25.850780+0000","last_peered":"2024-04-25T11:20:25.850780+0000","last_clean":"2024-04-25T11:20:25.850780+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:25.850780+0000","last_undegraded":"2024-04-25T11:20:25.850780+0000","last_fullsized":"2024-04-25T11:20:25.850780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066095+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:20:25.066095+0000","last_peered":"2024-04-25T11:20:25.066095+0000","last_clean":"2024-04-25T11:20:25.066095+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:20:25.066095+0000","last_undegraded":"2024-04-25T11:20:25.066095+0000","last_fullsized":"2024-04-25T11:20:25.066095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850239+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:25.850239+0000","last_peered":"2024-04-25T11:20:25.850239+0000","last_clean":"2024-04-25T11:20:25.850239+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:25.850239+0000","last_undegraded":"2024-04-25T11:20:25.850239+0000","last_fullsized":"2024-04-25T11:20:25.850239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848175+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:25.848175+0000","last_peered":"2024-04-25T11:20:25.848175+0000","last_clean":"2024-04-25T11:20:25.848175+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:25.848175+0000","last_undegraded":"2024-04-25T11:20:25.848175+0000","last_fullsized":"2024-04-25T11:20:25.848175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849105+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:20:25.849105+0000","last_peered":"2024-04-25T11:20:25.849105+0000","last_clean":"2024-04-25T11:20:25.849105+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:20:25.849105+0000","last_undegraded":"2024-04-25T11:20:25.849105+0000","last_fullsized":"2024-04-25T11:20:25.849105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849839+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:25.849839+0000","last_peered":"2024-04-25T11:20:25.849839+0000","last_clean":"2024-04-25T11:20:25.849839+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:25.849839+0000","last_undegraded":"2024-04-25T11:20:25.849839+0000","last_fullsized":"2024-04-25T11:20:25.849839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066978+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:25.066978+0000","last_peered":"2024-04-25T11:20:25.066978+0000","last_clean":"2024-04-25T11:20:25.066978+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:25.066978+0000","last_undegraded":"2024-04-25T11:20:25.066978+0000","last_fullsized":"2024-04-25T11:20:25.066978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703462+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:20:25.703462+0000","last_peered":"2024-04-25T11:20:25.703462+0000","last_clean":"2024-04-25T11:20:25.703462+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:20:25.703462+0000","last_undegraded":"2024-04-25T11:20:25.703462+0000","last_fullsized":"2024-04-25T11:20:25.703462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849435+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:20:25.849435+0000","last_peered":"2024-04-25T11:20:25.849435+0000","last_clean":"2024-04-25T11:20:25.849435+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:20:25.849435+0000","last_undegraded":"2024-04-25T11:20:25.849435+0000","last_fullsized":"2024-04-25T11:20:25.849435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066131+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:25.066131+0000","last_peered":"2024-04-25T11:20:25.066131+0000","last_clean":"2024-04-25T11:20:25.066131+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:25.066131+0000","last_undegraded":"2024-04-25T11:20:25.066131+0000","last_fullsized":"2024-04-25T11:20:25.066131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852310+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:20:25.852310+0000","last_peered":"2024-04-25T11:20:25.852310+0000","last_clean":"2024-04-25T11:20:25.852310+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:20:25.852310+0000","last_undegraded":"2024-04-25T11:20:25.852310+0000","last_fullsized":"2024-04-25T11:20:25.852310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065502+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:25.065502+0000","last_peered":"2024-04-25T11:20:25.065502+0000","last_clean":"2024-04-25T11:20:25.065502+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:25.065502+0000","last_undegraded":"2024-04-25T11:20:25.065502+0000","last_fullsized":"2024-04-25T11:20:25.065502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703910+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:20:25.703910+0000","last_peered":"2024-04-25T11:20:25.703910+0000","last_clean":"2024-04-25T11:20:25.703910+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:20:25.703910+0000","last_undegraded":"2024-04-25T11:20:25.703910+0000","last_fullsized":"2024-04-25T11:20:25.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849639+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:20:25.849639+0000","last_peered":"2024-04-25T11:20:25.849639+0000","last_clean":"2024-04-25T11:20:25.849639+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:20:25.849639+0000","last_undegraded":"2024-04-25T11:20:25.849639+0000","last_fullsized":"2024-04-25T11:20:25.849639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849150+0000","last_change":"2024-04-25T11:20:24.098651+0000","last_active":"2024-04-25T11:20:25.849150+0000","last_peered":"2024-04-25T11:20:25.849150+0000","last_clean":"2024-04-25T11:20:25.849150+0000","last_became_active":"2024-04-25T11:20:24.097985+0000","last_became_peered":"2024-04-25T11:20:24.097985+0000","last_unstale":"2024-04-25T11:20:25.849150+0000","last_undegraded":"2024-04-25T11:20:25.849150+0000","last_fullsized":"2024-04-25T11:20:25.849150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852724+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:20:25.852724+0000","last_peered":"2024-04-25T11:20:25.852724+0000","last_clean":"2024-04-25T11:20:25.852724+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:20:25.852724+0000","last_undegraded":"2024-04-25T11:20:25.852724+0000","last_fullsized":"2024-04-25T11:20:25.852724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067792+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:20:25.067792+0000","last_peered":"2024-04-25T11:20:25.067792+0000","last_clean":"2024-04-25T11:20:25.067792+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:20:25.067792+0000","last_undegraded":"2024-04-25T11:20:25.067792+0000","last_fullsized":"2024-04-25T11:20:25.067792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702944+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:20:25.702944+0000","last_peered":"2024-04-25T11:20:25.702944+0000","last_clean":"2024-04-25T11:20:25.702944+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:20:25.702944+0000","last_undegraded":"2024-04-25T11:20:25.702944+0000","last_fullsized":"2024-04-25T11:20:25.702944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852444+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:20:25.852444+0000","last_peered":"2024-04-25T11:20:25.852444+0000","last_clean":"2024-04-25T11:20:25.852444+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:20:25.852444+0000","last_undegraded":"2024-04-25T11:20:25.852444+0000","last_fullsized":"2024-04-25T11:20:25.852444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851726+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:20:25.851726+0000","last_peered":"2024-04-25T11:20:25.851726+0000","last_clean":"2024-04-25T11:20:25.851726+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:20:25.851726+0000","last_undegraded":"2024-04-25T11:20:25.851726+0000","last_fullsized":"2024-04-25T11:20:25.851726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850209+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:20:25.850209+0000","last_peered":"2024-04-25T11:20:25.850209+0000","last_clean":"2024-04-25T11:20:25.850209+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:20:25.850209+0000","last_undegraded":"2024-04-25T11:20:25.850209+0000","last_fullsized":"2024-04-25T11:20:25.850209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849636+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:20:25.849636+0000","last_peered":"2024-04-25T11:20:25.849636+0000","last_clean":"2024-04-25T11:20:25.849636+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:20:25.849636+0000","last_undegraded":"2024-04-25T11:20:25.849636+0000","last_fullsized":"2024-04-25T11:20:25.849636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065250+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:20:25.065250+0000","last_peered":"2024-04-25T11:20:25.065250+0000","last_clean":"2024-04-25T11:20:25.065250+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:20:25.065250+0000","last_undegraded":"2024-04-25T11:20:25.065250+0000","last_fullsized":"2024-04-25T11:20:25.065250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849363+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:25.849363+0000","last_peered":"2024-04-25T11:20:25.849363+0000","last_clean":"2024-04-25T11:20:25.849363+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:25.849363+0000","last_undegraded":"2024-04-25T11:20:25.849363+0000","last_fullsized":"2024-04-25T11:20:25.849363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065441+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:25.065441+0000","last_peered":"2024-04-25T11:20:25.065441+0000","last_clean":"2024-04-25T11:20:25.065441+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:25.065441+0000","last_undegraded":"2024-04-25T11:20:25.065441+0000","last_fullsized":"2024-04-25T11:20:25.065441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065600+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:20:25.065600+0000","last_peered":"2024-04-25T11:20:25.065600+0000","last_clean":"2024-04-25T11:20:25.065600+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:20:25.065600+0000","last_undegraded":"2024-04-25T11:20:25.065600+0000","last_fullsized":"2024-04-25T11:20:25.065600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071194+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:25.071194+0000","last_peered":"2024-04-25T11:20:25.071194+0000","last_clean":"2024-04-25T11:20:25.071194+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:25.071194+0000","last_undegraded":"2024-04-25T11:20:25.071194+0000","last_fullsized":"2024-04-25T11:20:25.071194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850312+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:20:25.850312+0000","last_peered":"2024-04-25T11:20:25.850312+0000","last_clean":"2024-04-25T11:20:25.850312+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:20:25.850312+0000","last_undegraded":"2024-04-25T11:20:25.850312+0000","last_fullsized":"2024-04-25T11:20:25.850312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703560+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:20:25.703560+0000","last_peered":"2024-04-25T11:20:25.703560+0000","last_clean":"2024-04-25T11:20:25.703560+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:20:25.703560+0000","last_undegraded":"2024-04-25T11:20:25.703560+0000","last_fullsized":"2024-04-25T11:20:25.703560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067907+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:25.067907+0000","last_peered":"2024-04-25T11:20:25.067907+0000","last_clean":"2024-04-25T11:20:25.067907+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:25.067907+0000","last_undegraded":"2024-04-25T11:20:25.067907+0000","last_fullsized":"2024-04-25T11:20:25.067907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849479+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:25.849479+0000","last_peered":"2024-04-25T11:20:25.849479+0000","last_clean":"2024-04-25T11:20:25.849479+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:25.849479+0000","last_undegraded":"2024-04-25T11:20:25.849479+0000","last_fullsized":"2024-04-25T11:20:25.849479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849577+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:20:25.849577+0000","last_peered":"2024-04-25T11:20:25.849577+0000","last_clean":"2024-04-25T11:20:25.849577+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:20:25.849577+0000","last_undegraded":"2024-04-25T11:20:25.849577+0000","last_fullsized":"2024-04-25T11:20:25.849577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704488+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:20:25.704488+0000","last_peered":"2024-04-25T11:20:25.704488+0000","last_clean":"2024-04-25T11:20:25.704488+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:20:25.704488+0000","last_undegraded":"2024-04-25T11:20:25.704488+0000","last_fullsized":"2024-04-25T11:20:25.704488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851990+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:20:25.851990+0000","last_peered":"2024-04-25T11:20:25.851990+0000","last_clean":"2024-04-25T11:20:25.851990+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:20:25.851990+0000","last_undegraded":"2024-04-25T11:20:25.851990+0000","last_fullsized":"2024-04-25T11:20:25.851990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:41.996974+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:20:41.996974+0000","last_peered":"2024-04-25T11:20:41.996974+0000","last_clean":"2024-04-25T11:20:41.996974+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:20:41.996974+0000","last_undegraded":"2024-04-25T11:20:41.996974+0000","last_fullsized":"2024-04-25T11:20:41.996974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849162+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:25.849162+0000","last_peered":"2024-04-25T11:20:25.849162+0000","last_clean":"2024-04-25T11:20:25.849162+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:25.849162+0000","last_undegraded":"2024-04-25T11:20:25.849162+0000","last_fullsized":"2024-04-25T11:20:25.849162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851639+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:20:25.851639+0000","last_peered":"2024-04-25T11:20:25.851639+0000","last_clean":"2024-04-25T11:20:25.851639+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:20:25.851639+0000","last_undegraded":"2024-04-25T11:20:25.851639+0000","last_fullsized":"2024-04-25T11:20:25.851639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849514+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:25.849514+0000","last_peered":"2024-04-25T11:20:25.849514+0000","last_clean":"2024-04-25T11:20:25.849514+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:25.849514+0000","last_undegraded":"2024-04-25T11:20:25.849514+0000","last_fullsized":"2024-04-25T11:20:25.849514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852817+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:20:25.852817+0000","last_peered":"2024-04-25T11:20:25.852817+0000","last_clean":"2024-04-25T11:20:25.852817+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:20:25.852817+0000","last_undegraded":"2024-04-25T11:20:25.852817+0000","last_fullsized":"2024-04-25T11:20:25.852817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848647+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:25.848647+0000","last_peered":"2024-04-25T11:20:25.848647+0000","last_clean":"2024-04-25T11:20:25.848647+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:25.848647+0000","last_undegraded":"2024-04-25T11:20:25.848647+0000","last_fullsized":"2024-04-25T11:20:25.848647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704293+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:20:25.704293+0000","last_peered":"2024-04-25T11:20:25.704293+0000","last_clean":"2024-04-25T11:20:25.704293+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:20:25.704293+0000","last_undegraded":"2024-04-25T11:20:25.704293+0000","last_fullsized":"2024-04-25T11:20:25.704293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066858+0000","last_change":"2024-04-25T11:20:24.093677+0000","last_active":"2024-04-25T11:20:25.066858+0000","last_peered":"2024-04-25T11:20:25.066858+0000","last_clean":"2024-04-25T11:20:25.066858+0000","last_became_active":"2024-04-25T11:20:24.092907+0000","last_became_peered":"2024-04-25T11:20:24.092907+0000","last_unstale":"2024-04-25T11:20:25.066858+0000","last_undegraded":"2024-04-25T11:20:25.066858+0000","last_fullsized":"2024-04-25T11:20:25.066858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:30.768495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704129+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:20:25.704129+0000","last_peered":"2024-04-25T11:20:25.704129+0000","last_clean":"2024-04-25T11:20:25.704129+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:20:25.704129+0000","last_undegraded":"2024-04-25T11:20:25.704129+0000","last_fullsized":"2024-04-25T11:20:25.704129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066326+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:20:25.066326+0000","last_peered":"2024-04-25T11:20:25.066326+0000","last_clean":"2024-04-25T11:20:25.066326+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:20:25.066326+0000","last_undegraded":"2024-04-25T11:20:25.066326+0000","last_fullsized":"2024-04-25T11:20:25.066326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067316+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:25.067316+0000","last_peered":"2024-04-25T11:20:25.067316+0000","last_clean":"2024-04-25T11:20:25.067316+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:25.067316+0000","last_undegraded":"2024-04-25T11:20:25.067316+0000","last_fullsized":"2024-04-25T11:20:25.067316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848888+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:25.848888+0000","last_peered":"2024-04-25T11:20:25.848888+0000","last_clean":"2024-04-25T11:20:25.848888+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:25.848888+0000","last_undegraded":"2024-04-25T11:20:25.848888+0000","last_fullsized":"2024-04-25T11:20:25.848888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848177+0000","last_change":"2024-04-25T11:20:20.040078+0000","last_active":"2024-04-25T11:20:25.848177+0000","last_peered":"2024-04-25T11:20:25.848177+0000","last_clean":"2024-04-25T11:20:25.848177+0000","last_became_active":"2024-04-25T11:20:20.039255+0000","last_became_peered":"2024-04-25T11:20:20.039255+0000","last_unstale":"2024-04-25T11:20:25.848177+0000","last_undegraded":"2024-04-25T11:20:25.848177+0000","last_fullsized":"2024-04-25T11:20:25.848177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703263+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:20:25.703263+0000","last_peered":"2024-04-25T11:20:25.703263+0000","last_clean":"2024-04-25T11:20:25.703263+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:20:25.703263+0000","last_undegraded":"2024-04-25T11:20:25.703263+0000","last_fullsized":"2024-04-25T11:20:25.703263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066138+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:25.066138+0000","last_peered":"2024-04-25T11:20:25.066138+0000","last_clean":"2024-04-25T11:20:25.066138+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:25.066138+0000","last_undegraded":"2024-04-25T11:20:25.066138+0000","last_fullsized":"2024-04-25T11:20:25.066138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066710+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:25.066710+0000","last_peered":"2024-04-25T11:20:25.066710+0000","last_clean":"2024-04-25T11:20:25.066710+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:25.066710+0000","last_undegraded":"2024-04-25T11:20:25.066710+0000","last_fullsized":"2024-04-25T11:20:25.066710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067819+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:20:25.067819+0000","last_peered":"2024-04-25T11:20:25.067819+0000","last_clean":"2024-04-25T11:20:25.067819+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:20:25.067819+0000","last_undegraded":"2024-04-25T11:20:25.067819+0000","last_fullsized":"2024-04-25T11:20:25.067819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065961+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:20:25.065961+0000","last_peered":"2024-04-25T11:20:25.065961+0000","last_clean":"2024-04-25T11:20:25.065961+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:20:25.065961+0000","last_undegraded":"2024-04-25T11:20:25.065961+0000","last_fullsized":"2024-04-25T11:20:25.065961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849428+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:25.849428+0000","last_peered":"2024-04-25T11:20:25.849428+0000","last_clean":"2024-04-25T11:20:25.849428+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:25.849428+0000","last_undegraded":"2024-04-25T11:20:25.849428+0000","last_fullsized":"2024-04-25T11:20:25.849428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066149+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:25.066149+0000","last_peered":"2024-04-25T11:20:25.066149+0000","last_clean":"2024-04-25T11:20:25.066149+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:25.066149+0000","last_undegraded":"2024-04-25T11:20:25.066149+0000","last_fullsized":"2024-04-25T11:20:25.066149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067406+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:25.067406+0000","last_peered":"2024-04-25T11:20:25.067406+0000","last_clean":"2024-04-25T11:20:25.067406+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:25.067406+0000","last_undegraded":"2024-04-25T11:20:25.067406+0000","last_fullsized":"2024-04-25T11:20:25.067406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850698+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:20:25.850698+0000","last_peered":"2024-04-25T11:20:25.850698+0000","last_clean":"2024-04-25T11:20:25.850698+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:20:25.850698+0000","last_undegraded":"2024-04-25T11:20:25.850698+0000","last_fullsized":"2024-04-25T11:20:25.850698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070455+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:25.070455+0000","last_peered":"2024-04-25T11:20:25.070455+0000","last_clean":"2024-04-25T11:20:25.070455+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:25.070455+0000","last_undegraded":"2024-04-25T11:20:25.070455+0000","last_fullsized":"2024-04-25T11:20:25.070455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065240+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:25.065240+0000","last_peered":"2024-04-25T11:20:25.065240+0000","last_clean":"2024-04-25T11:20:25.065240+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:25.065240+0000","last_undegraded":"2024-04-25T11:20:25.065240+0000","last_fullsized":"2024-04-25T11:20:25.065240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065744+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:25.065744+0000","last_peered":"2024-04-25T11:20:25.065744+0000","last_clean":"2024-04-25T11:20:25.065744+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:25.065744+0000","last_undegraded":"2024-04-25T11:20:25.065744+0000","last_fullsized":"2024-04-25T11:20:25.065744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703495+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:20:25.703495+0000","last_peered":"2024-04-25T11:20:25.703495+0000","last_clean":"2024-04-25T11:20:25.703495+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:20:25.703495+0000","last_undegraded":"2024-04-25T11:20:25.703495+0000","last_fullsized":"2024-04-25T11:20:25.703495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849293+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:25.849293+0000","last_peered":"2024-04-25T11:20:25.849293+0000","last_clean":"2024-04-25T11:20:25.849293+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:25.849293+0000","last_undegraded":"2024-04-25T11:20:25.849293+0000","last_fullsized":"2024-04-25T11:20:25.849293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.068272+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:25.068272+0000","last_peered":"2024-04-25T11:20:25.068272+0000","last_clean":"2024-04-25T11:20:25.068272+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:25.068272+0000","last_undegraded":"2024-04-25T11:20:25.068272+0000","last_fullsized":"2024-04-25T11:20:25.068272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849313+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:25.849313+0000","last_peered":"2024-04-25T11:20:25.849313+0000","last_clean":"2024-04-25T11:20:25.849313+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849313+0000","last_undegraded":"2024-04-25T11:20:25.849313+0000","last_fullsized":"2024-04-25T11:20:25.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066579+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:25.066579+0000","last_peered":"2024-04-25T11:20:25.066579+0000","last_clean":"2024-04-25T11:20:25.066579+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:25.066579+0000","last_undegraded":"2024-04-25T11:20:25.066579+0000","last_fullsized":"2024-04-25T11:20:25.066579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070896+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:25.070896+0000","last_peered":"2024-04-25T11:20:25.070896+0000","last_clean":"2024-04-25T11:20:25.070896+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:25.070896+0000","last_undegraded":"2024-04-25T11:20:25.070896+0000","last_fullsized":"2024-04-25T11:20:25.070896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703122+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:20:25.703122+0000","last_peered":"2024-04-25T11:20:25.703122+0000","last_clean":"2024-04-25T11:20:25.703122+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:20:25.703122+0000","last_undegraded":"2024-04-25T11:20:25.703122+0000","last_fullsized":"2024-04-25T11:20:25.703122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067675+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:25.067675+0000","last_peered":"2024-04-25T11:20:25.067675+0000","last_clean":"2024-04-25T11:20:25.067675+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:25.067675+0000","last_undegraded":"2024-04-25T11:20:25.067675+0000","last_fullsized":"2024-04-25T11:20:25.067675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703259+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:20:25.703259+0000","last_peered":"2024-04-25T11:20:25.703259+0000","last_clean":"2024-04-25T11:20:25.703259+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:20:25.703259+0000","last_undegraded":"2024-04-25T11:20:25.703259+0000","last_fullsized":"2024-04-25T11:20:25.703259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848929+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:25.848929+0000","last_peered":"2024-04-25T11:20:25.848929+0000","last_clean":"2024-04-25T11:20:25.848929+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:25.848929+0000","last_undegraded":"2024-04-25T11:20:25.848929+0000","last_fullsized":"2024-04-25T11:20:25.848929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066246+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:25.066246+0000","last_peered":"2024-04-25T11:20:25.066246+0000","last_clean":"2024-04-25T11:20:25.066246+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:25.066246+0000","last_undegraded":"2024-04-25T11:20:25.066246+0000","last_fullsized":"2024-04-25T11:20:25.066246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853167+0000","last_change":"2024-04-25T11:20:20.039980+0000","last_active":"2024-04-25T11:20:25.853167+0000","last_peered":"2024-04-25T11:20:25.853167+0000","last_clean":"2024-04-25T11:20:25.853167+0000","last_became_active":"2024-04-25T11:20:20.039297+0000","last_became_peered":"2024-04-25T11:20:20.039297+0000","last_unstale":"2024-04-25T11:20:25.853167+0000","last_undegraded":"2024-04-25T11:20:25.853167+0000","last_fullsized":"2024-04-25T11:20:25.853167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:18.780733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067502+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:25.067502+0000","last_peered":"2024-04-25T11:20:25.067502+0000","last_clean":"2024-04-25T11:20:25.067502+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:25.067502+0000","last_undegraded":"2024-04-25T11:20:25.067502+0000","last_fullsized":"2024-04-25T11:20:25.067502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704854+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:20:25.704854+0000","last_peered":"2024-04-25T11:20:25.704854+0000","last_clean":"2024-04-25T11:20:25.704854+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:20:25.704854+0000","last_undegraded":"2024-04-25T11:20:25.704854+0000","last_fullsized":"2024-04-25T11:20:25.704854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702747+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.702747+0000","last_peered":"2024-04-25T11:20:25.702747+0000","last_clean":"2024-04-25T11:20:25.702747+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:20:25.702747+0000","last_undegraded":"2024-04-25T11:20:25.702747+0000","last_fullsized":"2024-04-25T11:20:25.702747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848256+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:25.848256+0000","last_peered":"2024-04-25T11:20:25.848256+0000","last_clean":"2024-04-25T11:20:25.848256+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:25.848256+0000","last_undegraded":"2024-04-25T11:20:25.848256+0000","last_fullsized":"2024-04-25T11:20:25.848256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849356+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:25.849356+0000","last_peered":"2024-04-25T11:20:25.849356+0000","last_clean":"2024-04-25T11:20:25.849356+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:25.849356+0000","last_undegraded":"2024-04-25T11:20:25.849356+0000","last_fullsized":"2024-04-25T11:20:25.849356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067076+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:25.067076+0000","last_peered":"2024-04-25T11:20:25.067076+0000","last_clean":"2024-04-25T11:20:25.067076+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:25.067076+0000","last_undegraded":"2024-04-25T11:20:25.067076+0000","last_fullsized":"2024-04-25T11:20:25.067076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851339+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:20:25.851339+0000","last_peered":"2024-04-25T11:20:25.851339+0000","last_clean":"2024-04-25T11:20:25.851339+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:20:25.851339+0000","last_undegraded":"2024-04-25T11:20:25.851339+0000","last_fullsized":"2024-04-25T11:20:25.851339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065892+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:25.065892+0000","last_peered":"2024-04-25T11:20:25.065892+0000","last_clean":"2024-04-25T11:20:25.065892+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:25.065892+0000","last_undegraded":"2024-04-25T11:20:25.065892+0000","last_fullsized":"2024-04-25T11:20:25.065892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852736+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:20:25.852736+0000","last_peered":"2024-04-25T11:20:25.852736+0000","last_clean":"2024-04-25T11:20:25.852736+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:20:25.852736+0000","last_undegraded":"2024-04-25T11:20:25.852736+0000","last_fullsized":"2024-04-25T11:20:25.852736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066385+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:25.066385+0000","last_peered":"2024-04-25T11:20:25.066385+0000","last_clean":"2024-04-25T11:20:25.066385+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:25.066385+0000","last_undegraded":"2024-04-25T11:20:25.066385+0000","last_fullsized":"2024-04-25T11:20:25.066385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849276+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:20:25.849276+0000","last_peered":"2024-04-25T11:20:25.849276+0000","last_clean":"2024-04-25T11:20:25.849276+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849276+0000","last_undegraded":"2024-04-25T11:20:25.849276+0000","last_fullsized":"2024-04-25T11:20:25.849276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703124+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:20:25.703124+0000","last_peered":"2024-04-25T11:20:25.703124+0000","last_clean":"2024-04-25T11:20:25.703124+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:20:25.703124+0000","last_undegraded":"2024-04-25T11:20:25.703124+0000","last_fullsized":"2024-04-25T11:20:25.703124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849275+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:20:25.849275+0000","last_peered":"2024-04-25T11:20:25.849275+0000","last_clean":"2024-04-25T11:20:25.849275+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:20:25.849275+0000","last_undegraded":"2024-04-25T11:20:25.849275+0000","last_fullsized":"2024-04-25T11:20:25.849275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703709+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:20:25.703709+0000","last_peered":"2024-04-25T11:20:25.703709+0000","last_clean":"2024-04-25T11:20:25.703709+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:20:25.703709+0000","last_undegraded":"2024-04-25T11:20:25.703709+0000","last_fullsized":"2024-04-25T11:20:25.703709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849944+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:25.849944+0000","last_peered":"2024-04-25T11:20:25.849944+0000","last_clean":"2024-04-25T11:20:25.849944+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:25.849944+0000","last_undegraded":"2024-04-25T11:20:25.849944+0000","last_fullsized":"2024-04-25T11:20:25.849944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852390+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:20:25.852390+0000","last_peered":"2024-04-25T11:20:25.852390+0000","last_clean":"2024-04-25T11:20:25.852390+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:20:25.852390+0000","last_undegraded":"2024-04-25T11:20:25.852390+0000","last_fullsized":"2024-04-25T11:20:25.852390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:20:54.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:20:54.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 Processor -- start 2024-04-25T11:20:54.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 -- start start 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a540b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a540b4480 con 0x7f0a5415c480 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a5415c990 con 0x7f0a5415c070 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a5a168640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540b4a90 con 0x7f0a540b5810 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a53fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0a5415c480 0x7f0a54166910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53324/0 (socket says 172.21.15.136:53324) 2024-04-25T11:20:54.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a53fff640 1 -- 172.21.15.136:0/1210134834 learned_addr learned my addr 172.21.15.136:0/1210134834 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a537fe640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a540b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606888437 0 0) 0x7f0a540b4480 con 0x7f0a5415c480 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a3c003530 con 0x7f0a5415c480 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 549230237 0 0) 0x7f0a3c003530 con 0x7f0a5415c480 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 msgr2=0x7f0a540b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a540b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 >> v1:172.21.15.157:6789/0 conn(0x7f0a5415c070 legacy=0x7f0a5406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5415cc70 con 0x7f0a5415c480 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f0a4c001a00 con 0x7f0a5415c480 2024-04-25T11:20:54.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 >> v1:172.21.15.136:6789/0 conn(0x7f0a5415c480 legacy=0x7f0a54166910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a3c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a3c003f70 0x7f0a3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a3c004d50 0x7f0a3c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b4a90 con 0x7f0a3c004d50 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3c00bbf0 con 0x7f0a3c003f70 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3c00bdf0 con 0x7f0a540b5810 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a53fff640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a3c004d50 0x7f0a3c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a52ffd640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a3c003f70 0x7f0a3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.188+0000 7f0a537fe640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a3c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 -- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 msgr2=0x7f0a3c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a540b5810 0x7f0a3c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 -- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a3c003f70 msgr2=0x7f0a3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a3c003f70 0x7f0a3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 -- 172.21.15.136:0/1210134834 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c003530 con 0x7f0a3c004d50 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a3c004d50 0x7f0a3c005160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a5408ee40 tx=0x7f0a4c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db75a927a595cf2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c00a4a0 con 0x7f0a3c004d50 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c0ad650 con 0x7f0a3c004d50 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a527fc640 1 -- 172.21.15.136:0/1210134834 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0ada40 con 0x7f0a3c004d50 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 -- 172.21.15.136:0/1210134834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a3c004d50 msgr2=0x7f0a3c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 --2- 172.21.15.136:0/1210134834 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a3c004d50 0x7f0a3c005160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0a5408ee40 tx=0x7f0a4c01d360 comp rx=0 tx=0).stop 2024-04-25T11:20:54.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a53fff640 1 -- 172.21.15.136:0/1210134834 reap_dead start 2024-04-25T11:20:54.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 -- 172.21.15.136:0/1210134834 shutdown_connections 2024-04-25T11:20:54.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 -- 172.21.15.136:0/1210134834 >> 172.21.15.136:0/1210134834 conn(0x7f0a540743a0 msgr2=0x7f0a540d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:54.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 -- 172.21.15.136:0/1210134834 shutdown_connections 2024-04-25T11:20:54.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.192+0000 7f0a5a168640 1 -- 172.21.15.136:0/1210134834 wait complete. 2024-04-25T11:20:54.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 Processor -- start 2024-04-25T11:20:54.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 -- start start 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 0x7f0a54155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a54155990 0x7f0a540ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5406e030 con 0x7f0a541f2b10 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5415cf50 con 0x7f0a541502b0 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a5a168640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5406dc20 con 0x7f0a54155990 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36970/0 (socket says 172.21.15.136:36970) 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 -- 172.21.15.136:0/4031228579 learned_addr learned my addr 172.21.15.136:0/4031228579 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a53fff640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a54155990 0x7f0a540ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a52ffd640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 0x7f0a54155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 -- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a54155990 msgr2=0x7f0a540ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a54155990 0x7f0a540ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 -- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 msgr2=0x7f0a54155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 0x7f0a54155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a537fe640 1 -- 172.21.15.136:0/4031228579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540c9fb0 con 0x7f0a541f2b10 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.196+0000 7f0a52ffd640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 0x7f0a54155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a537fe640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a440115e0 tx=0x7f0a44011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400f8f0 con 0x7f0a541f2b10 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4400fa90 con 0x7f0a541f2b10 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a540ca290 con 0x7f0a541f2b10 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4400fe40 con 0x7f0a541f2b10 2024-04-25T11:20:54.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a54154c50 con 0x7f0a541f2b10 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a5406a350 con 0x7f0a541f2b10 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0a44015460 con 0x7f0a541f2b10 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 0x7f0a40071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0a4408ff50 con 0x7f0a541f2b10 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a52ffd640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 0x7f0a40071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.200+0000 7f0a52ffd640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 0x7f0a40071910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a540c80f0 tx=0x7f0a48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.204+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0a44019030 con 0x7f0a541f2b10 2024-04-25T11:20:54.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.364+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a540cd5f0 con 0x7f0a4006f400 2024-04-25T11:20:54.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.388+0000 7f0a3bfff640 1 -- 172.21.15.136:0/4031228579 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332527 (secure 0 0 0) 0x7f0a540cd5f0 con 0x7f0a4006f400 2024-04-25T11:20:54.390 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:54.392 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:20:54.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 msgr2=0x7f0a40071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 0x7f0a40071910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a540c80f0 tx=0x7f0a48009040 comp rx=0 tx=0).stop 2024-04-25T11:20:54.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 msgr2=0x7f0a5414fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a440115e0 tx=0x7f0a44011aa0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 shutdown_connections 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0a4006f400 0x7f0a40071910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a54155990 0x7f0a540ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a541502b0 0x7f0a54155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 --2- 172.21.15.136:0/4031228579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a541f2b10 0x7f0a5414fd70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 >> 172.21.15.136:0/4031228579 conn(0x7f0a540743a0 msgr2=0x7f0a54075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 shutdown_connections 2024-04-25T11:20:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.396+0000 7f0a5a168640 1 -- 172.21.15.136:0/4031228579 wait complete. 2024-04-25T11:20:54.413 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-04-25T11:20:52.765071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":150,"num_write_kb":1647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236000,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738720,"statfs":{"total":773094113280,"available":772852449280,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005363"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070611+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:25.070611+0000","last_peered":"2024-04-25T11:20:25.070611+0000","last_clean":"2024-04-25T11:20:25.070611+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:25.070611+0000","last_undegraded":"2024-04-25T11:20:25.070611+0000","last_fullsized":"2024-04-25T11:20:25.070611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066703+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:25.066703+0000","last_peered":"2024-04-25T11:20:25.066703+0000","last_clean":"2024-04-25T11:20:25.066703+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:25.066703+0000","last_undegraded":"2024-04-25T11:20:25.066703+0000","last_fullsized":"2024-04-25T11:20:25.066703+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070831+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:25.070831+0000","last_peered":"2024-04-25T11:20:25.070831+0000","last_clean":"2024-04-25T11:20:25.070831+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:25.070831+0000","last_undegraded":"2024-04-25T11:20:25.070831+0000","last_fullsized":"2024-04-25T11:20:25.070831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848527+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:25.848527+0000","last_peered":"2024-04-25T11:20:25.848527+0000","last_clean":"2024-04-25T11:20:25.848527+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:25.848527+0000","last_undegraded":"2024-04-25T11:20:25.848527+0000","last_fullsized":"2024-04-25T11:20:25.848527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850005+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:20:25.850005+0000","last_peered":"2024-04-25T11:20:25.850005+0000","last_clean":"2024-04-25T11:20:25.850005+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:20:25.850005+0000","last_undegraded":"2024-04-25T11:20:25.850005+0000","last_fullsized":"2024-04-25T11:20:25.850005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852853+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:20:25.852853+0000","last_peered":"2024-04-25T11:20:25.852853+0000","last_clean":"2024-04-25T11:20:25.852853+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.852853+0000","last_undegraded":"2024-04-25T11:20:25.852853+0000","last_fullsized":"2024-04-25T11:20:25.852853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703772+0000","last_change":"2024-04-25T11:20:20.032982+0000","last_active":"2024-04-25T11:20:25.703772+0000","last_peered":"2024-04-25T11:20:25.703772+0000","last_clean":"2024-04-25T11:20:25.703772+0000","last_became_active":"2024-04-25T11:20:20.032127+0000","last_became_peered":"2024-04-25T11:20:20.032127+0000","last_unstale":"2024-04-25T11:20:25.703772+0000","last_undegraded":"2024-04-25T11:20:25.703772+0000","last_fullsized":"2024-04-25T11:20:25.703772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852476+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:20:25.852476+0000","last_peered":"2024-04-25T11:20:25.852476+0000","last_clean":"2024-04-25T11:20:25.852476+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:20:25.852476+0000","last_undegraded":"2024-04-25T11:20:25.852476+0000","last_fullsized":"2024-04-25T11:20:25.852476+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851378+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:20:25.851378+0000","last_peered":"2024-04-25T11:20:25.851378+0000","last_clean":"2024-04-25T11:20:25.851378+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:20:25.851378+0000","last_undegraded":"2024-04-25T11:20:25.851378+0000","last_fullsized":"2024-04-25T11:20:25.851378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704317+0000","last_change":"2024-04-25T11:20:20.031174+0000","last_active":"2024-04-25T11:20:25.704317+0000","last_peered":"2024-04-25T11:20:25.704317+0000","last_clean":"2024-04-25T11:20:25.704317+0000","last_became_active":"2024-04-25T11:20:20.030340+0000","last_became_peered":"2024-04-25T11:20:20.030340+0000","last_unstale":"2024-04-25T11:20:25.704317+0000","last_undegraded":"2024-04-25T11:20:25.704317+0000","last_fullsized":"2024-04-25T11:20:25.704317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850069+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:20:25.850069+0000","last_peered":"2024-04-25T11:20:25.850069+0000","last_clean":"2024-04-25T11:20:25.850069+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:20:25.850069+0000","last_undegraded":"2024-04-25T11:20:25.850069+0000","last_fullsized":"2024-04-25T11:20:25.850069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065389+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:25.065389+0000","last_peered":"2024-04-25T11:20:25.065389+0000","last_clean":"2024-04-25T11:20:25.065389+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:25.065389+0000","last_undegraded":"2024-04-25T11:20:25.065389+0000","last_fullsized":"2024-04-25T11:20:25.065389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850811+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:20:25.850811+0000","last_peered":"2024-04-25T11:20:25.850811+0000","last_clean":"2024-04-25T11:20:25.850811+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:20:25.850811+0000","last_undegraded":"2024-04-25T11:20:25.850811+0000","last_fullsized":"2024-04-25T11:20:25.850811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851666+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:20:25.851666+0000","last_peered":"2024-04-25T11:20:25.851666+0000","last_clean":"2024-04-25T11:20:25.851666+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:20:25.851666+0000","last_undegraded":"2024-04-25T11:20:25.851666+0000","last_fullsized":"2024-04-25T11:20:25.851666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067265+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:25.067265+0000","last_peered":"2024-04-25T11:20:25.067265+0000","last_clean":"2024-04-25T11:20:25.067265+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:25.067265+0000","last_undegraded":"2024-04-25T11:20:25.067265+0000","last_fullsized":"2024-04-25T11:20:25.067265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850338+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:20:25.850338+0000","last_peered":"2024-04-25T11:20:25.850338+0000","last_clean":"2024-04-25T11:20:25.850338+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:20:25.850338+0000","last_undegraded":"2024-04-25T11:20:25.850338+0000","last_fullsized":"2024-04-25T11:20:25.850338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850575+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:20:25.850575+0000","last_peered":"2024-04-25T11:20:25.850575+0000","last_clean":"2024-04-25T11:20:25.850575+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:20:25.850575+0000","last_undegraded":"2024-04-25T11:20:25.850575+0000","last_fullsized":"2024-04-25T11:20:25.850575+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066984+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:25.066984+0000","last_peered":"2024-04-25T11:20:25.066984+0000","last_clean":"2024-04-25T11:20:25.066984+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:25.066984+0000","last_undegraded":"2024-04-25T11:20:25.066984+0000","last_fullsized":"2024-04-25T11:20:25.066984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848318+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:20:25.848318+0000","last_peered":"2024-04-25T11:20:25.848318+0000","last_clean":"2024-04-25T11:20:25.848318+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:20:25.848318+0000","last_undegraded":"2024-04-25T11:20:25.848318+0000","last_fullsized":"2024-04-25T11:20:25.848318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850089+0000","last_change":"2024-04-25T11:20:24.165156+0000","last_active":"2024-04-25T11:20:25.850089+0000","last_peered":"2024-04-25T11:20:25.850089+0000","last_clean":"2024-04-25T11:20:25.850089+0000","last_became_active":"2024-04-25T11:20:24.164587+0000","last_became_peered":"2024-04-25T11:20:24.164587+0000","last_unstale":"2024-04-25T11:20:25.850089+0000","last_undegraded":"2024-04-25T11:20:25.850089+0000","last_fullsized":"2024-04-25T11:20:25.850089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851436+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:20:25.851436+0000","last_peered":"2024-04-25T11:20:25.851436+0000","last_clean":"2024-04-25T11:20:25.851436+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:20:25.851436+0000","last_undegraded":"2024-04-25T11:20:25.851436+0000","last_fullsized":"2024-04-25T11:20:25.851436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849318+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:20:25.849318+0000","last_peered":"2024-04-25T11:20:25.849318+0000","last_clean":"2024-04-25T11:20:25.849318+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:20:25.849318+0000","last_undegraded":"2024-04-25T11:20:25.849318+0000","last_fullsized":"2024-04-25T11:20:25.849318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850973+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:20:25.850973+0000","last_peered":"2024-04-25T11:20:25.850973+0000","last_clean":"2024-04-25T11:20:25.850973+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:20:25.850973+0000","last_undegraded":"2024-04-25T11:20:25.850973+0000","last_fullsized":"2024-04-25T11:20:25.850973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849979+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:25.849979+0000","last_peered":"2024-04-25T11:20:25.849979+0000","last_clean":"2024-04-25T11:20:25.849979+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:25.849979+0000","last_undegraded":"2024-04-25T11:20:25.849979+0000","last_fullsized":"2024-04-25T11:20:25.849979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066481+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:20:25.066481+0000","last_peered":"2024-04-25T11:20:25.066481+0000","last_clean":"2024-04-25T11:20:25.066481+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:20:25.066481+0000","last_undegraded":"2024-04-25T11:20:25.066481+0000","last_fullsized":"2024-04-25T11:20:25.066481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850564+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:20:25.850564+0000","last_peered":"2024-04-25T11:20:25.850564+0000","last_clean":"2024-04-25T11:20:25.850564+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:20:25.850564+0000","last_undegraded":"2024-04-25T11:20:25.850564+0000","last_fullsized":"2024-04-25T11:20:25.850564+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071084+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:25.071084+0000","last_peered":"2024-04-25T11:20:25.071084+0000","last_clean":"2024-04-25T11:20:25.071084+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:25.071084+0000","last_undegraded":"2024-04-25T11:20:25.071084+0000","last_fullsized":"2024-04-25T11:20:25.071084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850091+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:20:25.850091+0000","last_peered":"2024-04-25T11:20:25.850091+0000","last_clean":"2024-04-25T11:20:25.850091+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:20:25.850091+0000","last_undegraded":"2024-04-25T11:20:25.850091+0000","last_fullsized":"2024-04-25T11:20:25.850091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066330+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:25.066330+0000","last_peered":"2024-04-25T11:20:25.066330+0000","last_clean":"2024-04-25T11:20:25.066330+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:25.066330+0000","last_undegraded":"2024-04-25T11:20:25.066330+0000","last_fullsized":"2024-04-25T11:20:25.066330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850885+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:25.850885+0000","last_peered":"2024-04-25T11:20:25.850885+0000","last_clean":"2024-04-25T11:20:25.850885+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:25.850885+0000","last_undegraded":"2024-04-25T11:20:25.850885+0000","last_fullsized":"2024-04-25T11:20:25.850885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853257+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:20:25.853257+0000","last_peered":"2024-04-25T11:20:25.853257+0000","last_clean":"2024-04-25T11:20:25.853257+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.853257+0000","last_undegraded":"2024-04-25T11:20:25.853257+0000","last_fullsized":"2024-04-25T11:20:25.853257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848798+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:25.848798+0000","last_peered":"2024-04-25T11:20:25.848798+0000","last_clean":"2024-04-25T11:20:25.848798+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:25.848798+0000","last_undegraded":"2024-04-25T11:20:25.848798+0000","last_fullsized":"2024-04-25T11:20:25.848798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704541+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.704541+0000","last_peered":"2024-04-25T11:20:25.704541+0000","last_clean":"2024-04-25T11:20:25.704541+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:20:25.704541+0000","last_undegraded":"2024-04-25T11:20:25.704541+0000","last_fullsized":"2024-04-25T11:20:25.704541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850957+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:25.850957+0000","last_peered":"2024-04-25T11:20:25.850957+0000","last_clean":"2024-04-25T11:20:25.850957+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:25.850957+0000","last_undegraded":"2024-04-25T11:20:25.850957+0000","last_fullsized":"2024-04-25T11:20:25.850957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848973+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:25.848973+0000","last_peered":"2024-04-25T11:20:25.848973+0000","last_clean":"2024-04-25T11:20:25.848973+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:25.848973+0000","last_undegraded":"2024-04-25T11:20:25.848973+0000","last_fullsized":"2024-04-25T11:20:25.848973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852994+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:20:25.852994+0000","last_peered":"2024-04-25T11:20:25.852994+0000","last_clean":"2024-04-25T11:20:25.852994+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:20:25.852994+0000","last_undegraded":"2024-04-25T11:20:25.852994+0000","last_fullsized":"2024-04-25T11:20:25.852994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703782+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:20:25.703782+0000","last_peered":"2024-04-25T11:20:25.703782+0000","last_clean":"2024-04-25T11:20:25.703782+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:20:25.703782+0000","last_undegraded":"2024-04-25T11:20:25.703782+0000","last_fullsized":"2024-04-25T11:20:25.703782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850807+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:25.850807+0000","last_peered":"2024-04-25T11:20:25.850807+0000","last_clean":"2024-04-25T11:20:25.850807+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:25.850807+0000","last_undegraded":"2024-04-25T11:20:25.850807+0000","last_fullsized":"2024-04-25T11:20:25.850807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850780+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:25.850780+0000","last_peered":"2024-04-25T11:20:25.850780+0000","last_clean":"2024-04-25T11:20:25.850780+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:25.850780+0000","last_undegraded":"2024-04-25T11:20:25.850780+0000","last_fullsized":"2024-04-25T11:20:25.850780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066095+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:20:25.066095+0000","last_peered":"2024-04-25T11:20:25.066095+0000","last_clean":"2024-04-25T11:20:25.066095+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:20:25.066095+0000","last_undegraded":"2024-04-25T11:20:25.066095+0000","last_fullsized":"2024-04-25T11:20:25.066095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850239+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:25.850239+0000","last_peered":"2024-04-25T11:20:25.850239+0000","last_clean":"2024-04-25T11:20:25.850239+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:25.850239+0000","last_undegraded":"2024-04-25T11:20:25.850239+0000","last_fullsized":"2024-04-25T11:20:25.850239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848175+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:25.848175+0000","last_peered":"2024-04-25T11:20:25.848175+0000","last_clean":"2024-04-25T11:20:25.848175+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:25.848175+0000","last_undegraded":"2024-04-25T11:20:25.848175+0000","last_fullsized":"2024-04-25T11:20:25.848175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849105+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:20:25.849105+0000","last_peered":"2024-04-25T11:20:25.849105+0000","last_clean":"2024-04-25T11:20:25.849105+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:20:25.849105+0000","last_undegraded":"2024-04-25T11:20:25.849105+0000","last_fullsized":"2024-04-25T11:20:25.849105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849839+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:25.849839+0000","last_peered":"2024-04-25T11:20:25.849839+0000","last_clean":"2024-04-25T11:20:25.849839+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:25.849839+0000","last_undegraded":"2024-04-25T11:20:25.849839+0000","last_fullsized":"2024-04-25T11:20:25.849839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066978+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:25.066978+0000","last_peered":"2024-04-25T11:20:25.066978+0000","last_clean":"2024-04-25T11:20:25.066978+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:25.066978+0000","last_undegraded":"2024-04-25T11:20:25.066978+0000","last_fullsized":"2024-04-25T11:20:25.066978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703462+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:20:25.703462+0000","last_peered":"2024-04-25T11:20:25.703462+0000","last_clean":"2024-04-25T11:20:25.703462+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:20:25.703462+0000","last_undegraded":"2024-04-25T11:20:25.703462+0000","last_fullsized":"2024-04-25T11:20:25.703462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849435+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:20:25.849435+0000","last_peered":"2024-04-25T11:20:25.849435+0000","last_clean":"2024-04-25T11:20:25.849435+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:20:25.849435+0000","last_undegraded":"2024-04-25T11:20:25.849435+0000","last_fullsized":"2024-04-25T11:20:25.849435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066131+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:25.066131+0000","last_peered":"2024-04-25T11:20:25.066131+0000","last_clean":"2024-04-25T11:20:25.066131+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:25.066131+0000","last_undegraded":"2024-04-25T11:20:25.066131+0000","last_fullsized":"2024-04-25T11:20:25.066131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852310+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:20:25.852310+0000","last_peered":"2024-04-25T11:20:25.852310+0000","last_clean":"2024-04-25T11:20:25.852310+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:20:25.852310+0000","last_undegraded":"2024-04-25T11:20:25.852310+0000","last_fullsized":"2024-04-25T11:20:25.852310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065502+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:25.065502+0000","last_peered":"2024-04-25T11:20:25.065502+0000","last_clean":"2024-04-25T11:20:25.065502+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:25.065502+0000","last_undegraded":"2024-04-25T11:20:25.065502+0000","last_fullsized":"2024-04-25T11:20:25.065502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703910+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:20:25.703910+0000","last_peered":"2024-04-25T11:20:25.703910+0000","last_clean":"2024-04-25T11:20:25.703910+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:20:25.703910+0000","last_undegraded":"2024-04-25T11:20:25.703910+0000","last_fullsized":"2024-04-25T11:20:25.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849639+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:20:25.849639+0000","last_peered":"2024-04-25T11:20:25.849639+0000","last_clean":"2024-04-25T11:20:25.849639+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:20:25.849639+0000","last_undegraded":"2024-04-25T11:20:25.849639+0000","last_fullsized":"2024-04-25T11:20:25.849639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849150+0000","last_change":"2024-04-25T11:20:24.098651+0000","last_active":"2024-04-25T11:20:25.849150+0000","last_peered":"2024-04-25T11:20:25.849150+0000","last_clean":"2024-04-25T11:20:25.849150+0000","last_became_active":"2024-04-25T11:20:24.097985+0000","last_became_peered":"2024-04-25T11:20:24.097985+0000","last_unstale":"2024-04-25T11:20:25.849150+0000","last_undegraded":"2024-04-25T11:20:25.849150+0000","last_fullsized":"2024-04-25T11:20:25.849150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852724+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:20:25.852724+0000","last_peered":"2024-04-25T11:20:25.852724+0000","last_clean":"2024-04-25T11:20:25.852724+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:20:25.852724+0000","last_undegraded":"2024-04-25T11:20:25.852724+0000","last_fullsized":"2024-04-25T11:20:25.852724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067792+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:20:25.067792+0000","last_peered":"2024-04-25T11:20:25.067792+0000","last_clean":"2024-04-25T11:20:25.067792+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:20:25.067792+0000","last_undegraded":"2024-04-25T11:20:25.067792+0000","last_fullsized":"2024-04-25T11:20:25.067792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702944+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:20:25.702944+0000","last_peered":"2024-04-25T11:20:25.702944+0000","last_clean":"2024-04-25T11:20:25.702944+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:20:25.702944+0000","last_undegraded":"2024-04-25T11:20:25.702944+0000","last_fullsized":"2024-04-25T11:20:25.702944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852444+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:20:25.852444+0000","last_peered":"2024-04-25T11:20:25.852444+0000","last_clean":"2024-04-25T11:20:25.852444+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:20:25.852444+0000","last_undegraded":"2024-04-25T11:20:25.852444+0000","last_fullsized":"2024-04-25T11:20:25.852444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851726+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:20:25.851726+0000","last_peered":"2024-04-25T11:20:25.851726+0000","last_clean":"2024-04-25T11:20:25.851726+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:20:25.851726+0000","last_undegraded":"2024-04-25T11:20:25.851726+0000","last_fullsized":"2024-04-25T11:20:25.851726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850209+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:20:25.850209+0000","last_peered":"2024-04-25T11:20:25.850209+0000","last_clean":"2024-04-25T11:20:25.850209+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:20:25.850209+0000","last_undegraded":"2024-04-25T11:20:25.850209+0000","last_fullsized":"2024-04-25T11:20:25.850209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849636+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:20:25.849636+0000","last_peered":"2024-04-25T11:20:25.849636+0000","last_clean":"2024-04-25T11:20:25.849636+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:20:25.849636+0000","last_undegraded":"2024-04-25T11:20:25.849636+0000","last_fullsized":"2024-04-25T11:20:25.849636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065250+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:20:25.065250+0000","last_peered":"2024-04-25T11:20:25.065250+0000","last_clean":"2024-04-25T11:20:25.065250+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:20:25.065250+0000","last_undegraded":"2024-04-25T11:20:25.065250+0000","last_fullsized":"2024-04-25T11:20:25.065250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849363+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:25.849363+0000","last_peered":"2024-04-25T11:20:25.849363+0000","last_clean":"2024-04-25T11:20:25.849363+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:25.849363+0000","last_undegraded":"2024-04-25T11:20:25.849363+0000","last_fullsized":"2024-04-25T11:20:25.849363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065441+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:25.065441+0000","last_peered":"2024-04-25T11:20:25.065441+0000","last_clean":"2024-04-25T11:20:25.065441+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:25.065441+0000","last_undegraded":"2024-04-25T11:20:25.065441+0000","last_fullsized":"2024-04-25T11:20:25.065441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065600+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:20:25.065600+0000","last_peered":"2024-04-25T11:20:25.065600+0000","last_clean":"2024-04-25T11:20:25.065600+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:20:25.065600+0000","last_undegraded":"2024-04-25T11:20:25.065600+0000","last_fullsized":"2024-04-25T11:20:25.065600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071194+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:25.071194+0000","last_peered":"2024-04-25T11:20:25.071194+0000","last_clean":"2024-04-25T11:20:25.071194+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:25.071194+0000","last_undegraded":"2024-04-25T11:20:25.071194+0000","last_fullsized":"2024-04-25T11:20:25.071194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850312+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:20:25.850312+0000","last_peered":"2024-04-25T11:20:25.850312+0000","last_clean":"2024-04-25T11:20:25.850312+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:20:25.850312+0000","last_undegraded":"2024-04-25T11:20:25.850312+0000","last_fullsized":"2024-04-25T11:20:25.850312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703560+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:20:25.703560+0000","last_peered":"2024-04-25T11:20:25.703560+0000","last_clean":"2024-04-25T11:20:25.703560+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:20:25.703560+0000","last_undegraded":"2024-04-25T11:20:25.703560+0000","last_fullsized":"2024-04-25T11:20:25.703560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067907+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:25.067907+0000","last_peered":"2024-04-25T11:20:25.067907+0000","last_clean":"2024-04-25T11:20:25.067907+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:25.067907+0000","last_undegraded":"2024-04-25T11:20:25.067907+0000","last_fullsized":"2024-04-25T11:20:25.067907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849479+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:25.849479+0000","last_peered":"2024-04-25T11:20:25.849479+0000","last_clean":"2024-04-25T11:20:25.849479+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:25.849479+0000","last_undegraded":"2024-04-25T11:20:25.849479+0000","last_fullsized":"2024-04-25T11:20:25.849479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849577+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:20:25.849577+0000","last_peered":"2024-04-25T11:20:25.849577+0000","last_clean":"2024-04-25T11:20:25.849577+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:20:25.849577+0000","last_undegraded":"2024-04-25T11:20:25.849577+0000","last_fullsized":"2024-04-25T11:20:25.849577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704488+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:20:25.704488+0000","last_peered":"2024-04-25T11:20:25.704488+0000","last_clean":"2024-04-25T11:20:25.704488+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:20:25.704488+0000","last_undegraded":"2024-04-25T11:20:25.704488+0000","last_fullsized":"2024-04-25T11:20:25.704488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851990+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:20:25.851990+0000","last_peered":"2024-04-25T11:20:25.851990+0000","last_clean":"2024-04-25T11:20:25.851990+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:20:25.851990+0000","last_undegraded":"2024-04-25T11:20:25.851990+0000","last_fullsized":"2024-04-25T11:20:25.851990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:41.996974+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:20:41.996974+0000","last_peered":"2024-04-25T11:20:41.996974+0000","last_clean":"2024-04-25T11:20:41.996974+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:20:41.996974+0000","last_undegraded":"2024-04-25T11:20:41.996974+0000","last_fullsized":"2024-04-25T11:20:41.996974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849162+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:25.849162+0000","last_peered":"2024-04-25T11:20:25.849162+0000","last_clean":"2024-04-25T11:20:25.849162+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:25.849162+0000","last_undegraded":"2024-04-25T11:20:25.849162+0000","last_fullsized":"2024-04-25T11:20:25.849162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851639+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:20:25.851639+0000","last_peered":"2024-04-25T11:20:25.851639+0000","last_clean":"2024-04-25T11:20:25.851639+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:20:25.851639+0000","last_undegraded":"2024-04-25T11:20:25.851639+0000","last_fullsized":"2024-04-25T11:20:25.851639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849514+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:25.849514+0000","last_peered":"2024-04-25T11:20:25.849514+0000","last_clean":"2024-04-25T11:20:25.849514+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:25.849514+0000","last_undegraded":"2024-04-25T11:20:25.849514+0000","last_fullsized":"2024-04-25T11:20:25.849514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852817+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:20:25.852817+0000","last_peered":"2024-04-25T11:20:25.852817+0000","last_clean":"2024-04-25T11:20:25.852817+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:20:25.852817+0000","last_undegraded":"2024-04-25T11:20:25.852817+0000","last_fullsized":"2024-04-25T11:20:25.852817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848647+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:25.848647+0000","last_peered":"2024-04-25T11:20:25.848647+0000","last_clean":"2024-04-25T11:20:25.848647+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:25.848647+0000","last_undegraded":"2024-04-25T11:20:25.848647+0000","last_fullsized":"2024-04-25T11:20:25.848647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704293+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:20:25.704293+0000","last_peered":"2024-04-25T11:20:25.704293+0000","last_clean":"2024-04-25T11:20:25.704293+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:20:25.704293+0000","last_undegraded":"2024-04-25T11:20:25.704293+0000","last_fullsized":"2024-04-25T11:20:25.704293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066858+0000","last_change":"2024-04-25T11:20:24.093677+0000","last_active":"2024-04-25T11:20:25.066858+0000","last_peered":"2024-04-25T11:20:25.066858+0000","last_clean":"2024-04-25T11:20:25.066858+0000","last_became_active":"2024-04-25T11:20:24.092907+0000","last_became_peered":"2024-04-25T11:20:24.092907+0000","last_unstale":"2024-04-25T11:20:25.066858+0000","last_undegraded":"2024-04-25T11:20:25.066858+0000","last_fullsized":"2024-04-25T11:20:25.066858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:30.768495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704129+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:20:25.704129+0000","last_peered":"2024-04-25T11:20:25.704129+0000","last_clean":"2024-04-25T11:20:25.704129+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:20:25.704129+0000","last_undegraded":"2024-04-25T11:20:25.704129+0000","last_fullsized":"2024-04-25T11:20:25.704129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066326+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:20:25.066326+0000","last_peered":"2024-04-25T11:20:25.066326+0000","last_clean":"2024-04-25T11:20:25.066326+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:20:25.066326+0000","last_undegraded":"2024-04-25T11:20:25.066326+0000","last_fullsized":"2024-04-25T11:20:25.066326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067316+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:25.067316+0000","last_peered":"2024-04-25T11:20:25.067316+0000","last_clean":"2024-04-25T11:20:25.067316+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:25.067316+0000","last_undegraded":"2024-04-25T11:20:25.067316+0000","last_fullsized":"2024-04-25T11:20:25.067316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848888+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:25.848888+0000","last_peered":"2024-04-25T11:20:25.848888+0000","last_clean":"2024-04-25T11:20:25.848888+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:25.848888+0000","last_undegraded":"2024-04-25T11:20:25.848888+0000","last_fullsized":"2024-04-25T11:20:25.848888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848177+0000","last_change":"2024-04-25T11:20:20.040078+0000","last_active":"2024-04-25T11:20:25.848177+0000","last_peered":"2024-04-25T11:20:25.848177+0000","last_clean":"2024-04-25T11:20:25.848177+0000","last_became_active":"2024-04-25T11:20:20.039255+0000","last_became_peered":"2024-04-25T11:20:20.039255+0000","last_unstale":"2024-04-25T11:20:25.848177+0000","last_undegraded":"2024-04-25T11:20:25.848177+0000","last_fullsized":"2024-04-25T11:20:25.848177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703263+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:20:25.703263+0000","last_peered":"2024-04-25T11:20:25.703263+0000","last_clean":"2024-04-25T11:20:25.703263+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:20:25.703263+0000","last_undegraded":"2024-04-25T11:20:25.703263+0000","last_fullsized":"2024-04-25T11:20:25.703263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066138+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:25.066138+0000","last_peered":"2024-04-25T11:20:25.066138+0000","last_clean":"2024-04-25T11:20:25.066138+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:25.066138+0000","last_undegraded":"2024-04-25T11:20:25.066138+0000","last_fullsized":"2024-04-25T11:20:25.066138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066710+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:25.066710+0000","last_peered":"2024-04-25T11:20:25.066710+0000","last_clean":"2024-04-25T11:20:25.066710+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:25.066710+0000","last_undegraded":"2024-04-25T11:20:25.066710+0000","last_fullsized":"2024-04-25T11:20:25.066710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067819+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:20:25.067819+0000","last_peered":"2024-04-25T11:20:25.067819+0000","last_clean":"2024-04-25T11:20:25.067819+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:20:25.067819+0000","last_undegraded":"2024-04-25T11:20:25.067819+0000","last_fullsized":"2024-04-25T11:20:25.067819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065961+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:20:25.065961+0000","last_peered":"2024-04-25T11:20:25.065961+0000","last_clean":"2024-04-25T11:20:25.065961+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:20:25.065961+0000","last_undegraded":"2024-04-25T11:20:25.065961+0000","last_fullsized":"2024-04-25T11:20:25.065961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849428+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:25.849428+0000","last_peered":"2024-04-25T11:20:25.849428+0000","last_clean":"2024-04-25T11:20:25.849428+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:25.849428+0000","last_undegraded":"2024-04-25T11:20:25.849428+0000","last_fullsized":"2024-04-25T11:20:25.849428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066149+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:25.066149+0000","last_peered":"2024-04-25T11:20:25.066149+0000","last_clean":"2024-04-25T11:20:25.066149+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:25.066149+0000","last_undegraded":"2024-04-25T11:20:25.066149+0000","last_fullsized":"2024-04-25T11:20:25.066149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067406+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:25.067406+0000","last_peered":"2024-04-25T11:20:25.067406+0000","last_clean":"2024-04-25T11:20:25.067406+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:25.067406+0000","last_undegraded":"2024-04-25T11:20:25.067406+0000","last_fullsized":"2024-04-25T11:20:25.067406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850698+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:20:25.850698+0000","last_peered":"2024-04-25T11:20:25.850698+0000","last_clean":"2024-04-25T11:20:25.850698+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:20:25.850698+0000","last_undegraded":"2024-04-25T11:20:25.850698+0000","last_fullsized":"2024-04-25T11:20:25.850698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070455+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:25.070455+0000","last_peered":"2024-04-25T11:20:25.070455+0000","last_clean":"2024-04-25T11:20:25.070455+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:25.070455+0000","last_undegraded":"2024-04-25T11:20:25.070455+0000","last_fullsized":"2024-04-25T11:20:25.070455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065240+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:25.065240+0000","last_peered":"2024-04-25T11:20:25.065240+0000","last_clean":"2024-04-25T11:20:25.065240+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:25.065240+0000","last_undegraded":"2024-04-25T11:20:25.065240+0000","last_fullsized":"2024-04-25T11:20:25.065240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065744+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:25.065744+0000","last_peered":"2024-04-25T11:20:25.065744+0000","last_clean":"2024-04-25T11:20:25.065744+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:25.065744+0000","last_undegraded":"2024-04-25T11:20:25.065744+0000","last_fullsized":"2024-04-25T11:20:25.065744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703495+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:20:25.703495+0000","last_peered":"2024-04-25T11:20:25.703495+0000","last_clean":"2024-04-25T11:20:25.703495+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:20:25.703495+0000","last_undegraded":"2024-04-25T11:20:25.703495+0000","last_fullsized":"2024-04-25T11:20:25.703495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849293+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:25.849293+0000","last_peered":"2024-04-25T11:20:25.849293+0000","last_clean":"2024-04-25T11:20:25.849293+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:25.849293+0000","last_undegraded":"2024-04-25T11:20:25.849293+0000","last_fullsized":"2024-04-25T11:20:25.849293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.068272+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:25.068272+0000","last_peered":"2024-04-25T11:20:25.068272+0000","last_clean":"2024-04-25T11:20:25.068272+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:25.068272+0000","last_undegraded":"2024-04-25T11:20:25.068272+0000","last_fullsized":"2024-04-25T11:20:25.068272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849313+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:25.849313+0000","last_peered":"2024-04-25T11:20:25.849313+0000","last_clean":"2024-04-25T11:20:25.849313+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849313+0000","last_undegraded":"2024-04-25T11:20:25.849313+0000","last_fullsized":"2024-04-25T11:20:25.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066579+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:25.066579+0000","last_peered":"2024-04-25T11:20:25.066579+0000","last_clean":"2024-04-25T11:20:25.066579+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:25.066579+0000","last_undegraded":"2024-04-25T11:20:25.066579+0000","last_fullsized":"2024-04-25T11:20:25.066579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070896+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:25.070896+0000","last_peered":"2024-04-25T11:20:25.070896+0000","last_clean":"2024-04-25T11:20:25.070896+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:25.070896+0000","last_undegraded":"2024-04-25T11:20:25.070896+0000","last_fullsized":"2024-04-25T11:20:25.070896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703122+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:20:25.703122+0000","last_peered":"2024-04-25T11:20:25.703122+0000","last_clean":"2024-04-25T11:20:25.703122+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:20:25.703122+0000","last_undegraded":"2024-04-25T11:20:25.703122+0000","last_fullsized":"2024-04-25T11:20:25.703122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067675+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:25.067675+0000","last_peered":"2024-04-25T11:20:25.067675+0000","last_clean":"2024-04-25T11:20:25.067675+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:25.067675+0000","last_undegraded":"2024-04-25T11:20:25.067675+0000","last_fullsized":"2024-04-25T11:20:25.067675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703259+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:20:25.703259+0000","last_peered":"2024-04-25T11:20:25.703259+0000","last_clean":"2024-04-25T11:20:25.703259+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:20:25.703259+0000","last_undegraded":"2024-04-25T11:20:25.703259+0000","last_fullsized":"2024-04-25T11:20:25.703259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848929+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:25.848929+0000","last_peered":"2024-04-25T11:20:25.848929+0000","last_clean":"2024-04-25T11:20:25.848929+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:25.848929+0000","last_undegraded":"2024-04-25T11:20:25.848929+0000","last_fullsized":"2024-04-25T11:20:25.848929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066246+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:25.066246+0000","last_peered":"2024-04-25T11:20:25.066246+0000","last_clean":"2024-04-25T11:20:25.066246+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:25.066246+0000","last_undegraded":"2024-04-25T11:20:25.066246+0000","last_fullsized":"2024-04-25T11:20:25.066246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853167+0000","last_change":"2024-04-25T11:20:20.039980+0000","last_active":"2024-04-25T11:20:25.853167+0000","last_peered":"2024-04-25T11:20:25.853167+0000","last_clean":"2024-04-25T11:20:25.853167+0000","last_became_active":"2024-04-25T11:20:20.039297+0000","last_became_peered":"2024-04-25T11:20:20.039297+0000","last_unstale":"2024-04-25T11:20:25.853167+0000","last_undegraded":"2024-04-25T11:20:25.853167+0000","last_fullsized":"2024-04-25T11:20:25.853167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:18.780733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067502+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:25.067502+0000","last_peered":"2024-04-25T11:20:25.067502+0000","last_clean":"2024-04-25T11:20:25.067502+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:25.067502+0000","last_undegraded":"2024-04-25T11:20:25.067502+0000","last_fullsized":"2024-04-25T11:20:25.067502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704854+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:20:25.704854+0000","last_peered":"2024-04-25T11:20:25.704854+0000","last_clean":"2024-04-25T11:20:25.704854+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:20:25.704854+0000","last_undegraded":"2024-04-25T11:20:25.704854+0000","last_fullsized":"2024-04-25T11:20:25.704854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702747+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.702747+0000","last_peered":"2024-04-25T11:20:25.702747+0000","last_clean":"2024-04-25T11:20:25.702747+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:20:25.702747+0000","last_undegraded":"2024-04-25T11:20:25.702747+0000","last_fullsized":"2024-04-25T11:20:25.702747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848256+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:25.848256+0000","last_peered":"2024-04-25T11:20:25.848256+0000","last_clean":"2024-04-25T11:20:25.848256+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:25.848256+0000","last_undegraded":"2024-04-25T11:20:25.848256+0000","last_fullsized":"2024-04-25T11:20:25.848256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849356+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:25.849356+0000","last_peered":"2024-04-25T11:20:25.849356+0000","last_clean":"2024-04-25T11:20:25.849356+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:25.849356+0000","last_undegraded":"2024-04-25T11:20:25.849356+0000","last_fullsized":"2024-04-25T11:20:25.849356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067076+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:25.067076+0000","last_peered":"2024-04-25T11:20:25.067076+0000","last_clean":"2024-04-25T11:20:25.067076+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:25.067076+0000","last_undegraded":"2024-04-25T11:20:25.067076+0000","last_fullsized":"2024-04-25T11:20:25.067076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851339+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:20:25.851339+0000","last_peered":"2024-04-25T11:20:25.851339+0000","last_clean":"2024-04-25T11:20:25.851339+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:20:25.851339+0000","last_undegraded":"2024-04-25T11:20:25.851339+0000","last_fullsized":"2024-04-25T11:20:25.851339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065892+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:25.065892+0000","last_peered":"2024-04-25T11:20:25.065892+0000","last_clean":"2024-04-25T11:20:25.065892+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:25.065892+0000","last_undegraded":"2024-04-25T11:20:25.065892+0000","last_fullsized":"2024-04-25T11:20:25.065892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852736+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:20:25.852736+0000","last_peered":"2024-04-25T11:20:25.852736+0000","last_clean":"2024-04-25T11:20:25.852736+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:20:25.852736+0000","last_undegraded":"2024-04-25T11:20:25.852736+0000","last_fullsized":"2024-04-25T11:20:25.852736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066385+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:25.066385+0000","last_peered":"2024-04-25T11:20:25.066385+0000","last_clean":"2024-04-25T11:20:25.066385+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:25.066385+0000","last_undegraded":"2024-04-25T11:20:25.066385+0000","last_fullsized":"2024-04-25T11:20:25.066385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849276+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:20:25.849276+0000","last_peered":"2024-04-25T11:20:25.849276+0000","last_clean":"2024-04-25T11:20:25.849276+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849276+0000","last_undegraded":"2024-04-25T11:20:25.849276+0000","last_fullsized":"2024-04-25T11:20:25.849276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703124+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:20:25.703124+0000","last_peered":"2024-04-25T11:20:25.703124+0000","last_clean":"2024-04-25T11:20:25.703124+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:20:25.703124+0000","last_undegraded":"2024-04-25T11:20:25.703124+0000","last_fullsized":"2024-04-25T11:20:25.703124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849275+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:20:25.849275+0000","last_peered":"2024-04-25T11:20:25.849275+0000","last_clean":"2024-04-25T11:20:25.849275+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:20:25.849275+0000","last_undegraded":"2024-04-25T11:20:25.849275+0000","last_fullsized":"2024-04-25T11:20:25.849275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703709+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:20:25.703709+0000","last_peered":"2024-04-25T11:20:25.703709+0000","last_clean":"2024-04-25T11:20:25.703709+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:20:25.703709+0000","last_undegraded":"2024-04-25T11:20:25.703709+0000","last_fullsized":"2024-04-25T11:20:25.703709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849944+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:25.849944+0000","last_peered":"2024-04-25T11:20:25.849944+0000","last_clean":"2024-04-25T11:20:25.849944+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:25.849944+0000","last_undegraded":"2024-04-25T11:20:25.849944+0000","last_fullsized":"2024-04-25T11:20:25.849944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852390+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:20:25.852390+0000","last_peered":"2024-04-25T11:20:25.852390+0000","last_clean":"2024-04-25T11:20:25.852390+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:20:25.852390+0000","last_undegraded":"2024-04-25T11:20:25.852390+0000","last_fullsized":"2024-04-25T11:20:25.852390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:20:54.421 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T11:20:54.422 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T11:20:54.422 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T11:20:54.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T11:20:54.423 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T11:20:54.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 Processor -- start 2024-04-25T11:20:54.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 -- start start 2024-04-25T11:20:54.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e58162660 0x7f9e58164a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e580c8090 con 0x7f9e58153190 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e580c6aa0 con 0x7f9e58158da0 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5e5f1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9e580c8370 con 0x7f9e58162660 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5cb8e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e58162660 0x7f9e58164a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e57fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9e58153190 0x7f9e580c9030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53332/0 (socket says 172.21.15.136:53332) 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e57fff640 1 -- 172.21.15.136:0/3419627634 learned_addr learned my addr 172.21.15.136:0/3419627634 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5cb8e640 1 -- 172.21.15.136:0/3419627634 >> v1:172.21.15.157:6789/0 conn(0x7f9e58158da0 legacy=0x7f9e581591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5cb8e640 1 -- 172.21.15.136:0/3419627634 >> v1:172.21.15.136:6789/0 conn(0x7f9e58153190 legacy=0x7f9e580c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.612+0000 7f9e5cb8e640 1 -- 172.21.15.136:0/3419627634 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e580c6d80 con 0x7f9e58162660 2024-04-25T11:20:54.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5cb8e640 1 --2- 172.21.15.136:0/3419627634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58162660 0x7f9e58164a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e48021360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef50f6959bad8ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e56ffd640 1 -- 172.21.15.136:0/3419627634 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480b6070 con 0x7f9e58162660 2024-04-25T11:20:54.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e56ffd640 1 -- 172.21.15.136:0/3419627634 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e480b1600 con 0x7f9e58162660 2024-04-25T11:20:54.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e56ffd640 1 -- 172.21.15.136:0/3419627634 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480aae60 con 0x7f9e58162660 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3419627634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58162660 msgr2=0x7f9e58164a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3419627634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58162660 0x7f9e58164a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e48021360 comp rx=0 tx=0).stop 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3419627634 shutdown_connections 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3419627634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58162660 0x7f9e58164a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3419627634 >> 172.21.15.136:0/3419627634 conn(0x7f9e580c9d50 msgr2=0x7f9e580ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3419627634 shutdown_connections 2024-04-25T11:20:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.616+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3419627634 wait complete. 2024-04-25T11:20:54.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 Processor -- start 2024-04-25T11:20:54.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- start start 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 0x7f9e580cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e58162660 0x7f9e5814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9e580ca870 con 0x7f9e58153190 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9e580c7340 con 0x7f9e58158da0 2024-04-25T11:20:54.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9e580ca9f0 con 0x7f9e58162660 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36986/0 (socket says 172.21.15.136:36986) 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 -- 172.21.15.136:0/3335007658 learned_addr learned my addr 172.21.15.136:0/3335007658 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5cb8e640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e58162660 0x7f9e5814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e577fe640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 0x7f9e580cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 -- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e58162660 msgr2=0x7f9e5814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e58162660 0x7f9e5814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 -- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 msgr2=0x7f9e580cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 0x7f9e580cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 -- 172.21.15.136:0/3335007658 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e580cc480 con 0x7f9e58153190 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e577fe640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 0x7f9e580cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e57fff640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c430 tx=0x7f9e5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017070 con 0x7f9e58153190 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e50005cf0 con 0x7f9e58153190 2024-04-25T11:20:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500072d0 con 0x7f9e58153190 2024-04-25T11:20:54.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e58152610 con 0x7f9e58153190 2024-04-25T11:20:54.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e580cb740 con 0x7f9e58153190 2024-04-25T11:20:54.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.620+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e5806a350 con 0x7f9e58153190 2024-04-25T11:20:54.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.624+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9e5001a070 con 0x7f9e58153190 2024-04-25T11:20:54.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.624+0000 7f9e54ff9640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 0x7f9e38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.624+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9e5008e7f0 con 0x7f9e58153190 2024-04-25T11:20:54.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.624+0000 7f9e577fe640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 0x7f9e38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.624+0000 7f9e577fe640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 0x7f9e38071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e5814d7c0 tx=0x7f9e40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:54.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.628+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9e5005ae70 con 0x7f9e58153190 2024-04-25T11:20:54.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.816+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f9e58073320 con 0x7f9e58153190 2024-04-25T11:20:54.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.824+0000 7f9e54ff9640 1 -- 172.21.15.136:0/3335007658 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+179548 (secure 0 0 0) 0x7f9e5005f170 con 0x7f9e58153190 2024-04-25T11:20:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:54.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 msgr2=0x7f9e38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 0x7f9e38071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e5814d7c0 tx=0x7f9e40009040 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 msgr2=0x7f9e581ef4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000c430 tx=0x7f9e5000c8f0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 shutdown_connections 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9e3806f400 0x7f9e38071910 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e58162660 0x7f9e5814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e58158da0 0x7f9e580cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 --2- 172.21.15.136:0/3335007658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e58153190 0x7f9e581ef4f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 >> 172.21.15.136:0/3335007658 conn(0x7f9e580c9d50 msgr2=0x7f9e5806e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 shutdown_connections 2024-04-25T11:20:54.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:54.828+0000 7f9e5e5f1640 1 -- 172.21.15.136:0/3335007658 wait complete. 2024-04-25T11:20:54.844 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":602770275},{"type":"v1","addr":"172.21.15.157:6833","nonce":602770275}]},"active_addr":"172.21.15.157:6833/602770275","active_change":"2024-04-25T11:19:58.733025+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":4294213131}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":684277787}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":2203292352}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.157:0","nonce":1227063183}]}]} 2024-04-25T11:20:54.848 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T11:20:54.848 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T11:20:54.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:54.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 Processor -- start 2024-04-25T11:20:54.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 -- start start 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f854c0cf490 0x7f854c0cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c06e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c15c410 con 0x7f854c156750 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f854c0cfde0 con 0x7f854c0cf490 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8550ca4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f854c15c6b0 con 0x7f854c06e470 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8549d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f854c0cf490 0x7f854c0cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f854ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c06e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f854ad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c06e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45048/0 (socket says 172.21.15.136:45048) 2024-04-25T11:20:54.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f854ad9d640 1 -- 172.21.15.136:0/3634682182 learned_addr learned my addr 172.21.15.136:0/3634682182 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8549d9b640 1 -- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 msgr2=0x7f854c06e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8549d9b640 1 --2- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c06e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8549d9b640 1 -- 172.21.15.136:0/3634682182 >> v1:172.21.15.157:6789/0 conn(0x7f854c156750 legacy=0x7f854c0ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.000+0000 7f8549d9b640 1 -- 172.21.15.136:0/3634682182 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c15c990 con 0x7f854c0cf490 2024-04-25T11:20:55.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8549d9b640 1 --2- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0cf490 0x7f854c0cf8a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8540009d90 tx=0x7f854001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b063724de0639d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f854959a640 1 -- 172.21.15.136:0/3634682182 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400b2070 con 0x7f854c0cf490 2024-04-25T11:20:55.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f854959a640 1 -- 172.21.15.136:0/3634682182 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85400a5830 con 0x7f854c0cf490 2024-04-25T11:20:55.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f854959a640 1 -- 172.21.15.136:0/3634682182 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400a5c00 con 0x7f854c0cf490 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 -- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0cf490 msgr2=0x7f854c0cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 --2- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0cf490 0x7f854c0cf8a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8540009d90 tx=0x7f854001c970 comp rx=0 tx=0).stop 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 -- 172.21.15.136:0/3634682182 shutdown_connections 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 --2- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c06e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 --2- 172.21.15.136:0/3634682182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0cf490 0x7f854c0cf8a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 -- 172.21.15.136:0/3634682182 >> 172.21.15.136:0/3634682182 conn(0x7f854c070f70 msgr2=0x7f854c068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 -- 172.21.15.136:0/3634682182 shutdown_connections 2024-04-25T11:20:55.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.004+0000 7f8550ca4640 1 -- 172.21.15.136:0/3634682182 wait complete. 2024-04-25T11:20:55.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 Processor -- start 2024-04-25T11:20:55.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- start start 2024-04-25T11:20:55.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 0x7f854c0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0cab20 con 0x7f854c0cf490 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0ca9a0 con 0x7f854c0b11d0 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f854c06d860 con 0x7f854c06e470 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34900/0 (socket says 172.21.15.136:34900) 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 -- 172.21.15.136:0/2675774497 learned_addr learned my addr 172.21.15.136:0/2675774497 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f854a59c640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f854ad9d640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 0x7f854c0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 -- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 msgr2=0x7f854c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c1580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 -- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 msgr2=0x7f854c0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 0x7f854c0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 -- 172.21.15.136:0/2675774497 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c15de60 con 0x7f854c0cf490 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f854ad9d640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 0x7f854c0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:55.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8549d9b640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85400b0040 tx=0x7f8540019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400b2050 con 0x7f854c0cf490 2024-04-25T11:20:55.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85400be070 con 0x7f854c0cf490 2024-04-25T11:20:55.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c1596e0 con 0x7f854c0cf490 2024-04-25T11:20:55.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400ad850 con 0x7f854c0cf490 2024-04-25T11:20:55.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c158e40 con 0x7f854c0cf490 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.008+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854c06a350 con 0x7f854c0cf490 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.012+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f85400ad9f0 con 0x7f854c0cf490 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.012+0000 7f853affd640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 0x7f852c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.012+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f854012a5c0 con 0x7f854c0cf490 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.012+0000 7f854a59c640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 0x7f852c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.012+0000 7f854a59c640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 0x7f852c071910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85440015b0 tx=0x7f8544009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.016+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8540016560 con 0x7f854c0cf490 2024-04-25T11:20:55.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.176+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f854c158bd0 con 0x7f854c0cf490 2024-04-25T11:20:55.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f853affd640 1 -- 172.21.15.136:0/2675774497 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7f85400ff030 con 0x7f854c0cf490 2024-04-25T11:20:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:55.177 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 msgr2=0x7f852c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 0x7f852c071910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85440015b0 tx=0x7f8544009040 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 msgr2=0x7f854c15d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85400b0040 tx=0x7f8540019710 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 shutdown_connections 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f854c0b11d0 0x7f854c0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f854c0cf490 0x7f854c15d920 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f852c06f400 0x7f852c071910 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 --2- 172.21.15.136:0/2675774497 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f854c06e470 0x7f854c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 >> 172.21.15.136:0/2675774497 conn(0x7f854c070f70 msgr2=0x7f854c1542b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 shutdown_connections 2024-04-25T11:20:55.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.180+0000 7f8550ca4640 1 -- 172.21.15.136:0/2675774497 wait complete. 2024-04-25T11:20:55.198 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T11:20:55.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:20:55.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.336+0000 7fc31d330640 1 Processor -- start 2024-04-25T11:20:55.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.336+0000 7fc31d330640 1 -- start start 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc318153190 0x7fc3180c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3180c9570 0x7fc318158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc3181592e0 0x7fc31815c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc3180c8370 con 0x7fc3180c9570 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc3181596f0 con 0x7fc3181592e0 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc31d330640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc3180c8090 con 0x7fc318153190 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3180c9570 0x7fc318158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3180c9570 0x7fc318158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34916/0 (socket says 172.21.15.136:34916) 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 -- 172.21.15.136:0/2746210328 learned_addr learned my addr 172.21.15.136:0/2746210328 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 -- 172.21.15.136:0/2746210328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc318153190 msgr2=0x7fc3180c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc316ffd640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc318153190 0x7fc3180c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc318153190 0x7fc3180c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 -- 172.21.15.136:0/2746210328 >> v2:172.21.15.157:3300/0 conn(0x7fc3181592e0 msgr2=0x7fc31815c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3177fe640 1 --2- 172.21.15.136:0/2746210328 >> v2:172.21.15.157:3300/0 conn(0x7fc3181592e0 0x7fc31815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 --2- 172.21.15.136:0/2746210328 >> v2:172.21.15.157:3300/0 conn(0x7fc3181592e0 0x7fc31815c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 -- 172.21.15.136:0/2746210328 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3180c6aa0 con 0x7fc3180c9570 2024-04-25T11:20:55.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc3167fc640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3180c9570 0x7fc318158da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc318072950 tx=0x7fc30c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb44f52aeba3f6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc315ffb640 1 -- 172.21.15.136:0/2746210328 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0b5070 con 0x7fc3180c9570 2024-04-25T11:20:55.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc315ffb640 1 -- 172.21.15.136:0/2746210328 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30c0a89d0 con 0x7fc3180c9570 2024-04-25T11:20:55.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.340+0000 7fc315ffb640 1 -- 172.21.15.136:0/2746210328 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0a8da0 con 0x7fc3180c9570 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- 172.21.15.136:0/2746210328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3180c9570 msgr2=0x7fc318158da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3180c9570 0x7fc318158da0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc318072950 tx=0x7fc30c01ed10 comp rx=0 tx=0).stop 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- 172.21.15.136:0/2746210328 shutdown_connections 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3180c9570 0x7fc318158da0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 --2- 172.21.15.136:0/2746210328 >> v2:172.21.15.157:3300/0 conn(0x7fc3181592e0 0x7fc31815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 --2- 172.21.15.136:0/2746210328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc318153190 0x7fc3180c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- 172.21.15.136:0/2746210328 >> 172.21.15.136:0/2746210328 conn(0x7fc3180c9d50 msgr2=0x7fc3180ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- 172.21.15.136:0/2746210328 shutdown_connections 2024-04-25T11:20:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- 172.21.15.136:0/2746210328 wait complete. 2024-04-25T11:20:55.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 Processor -- start 2024-04-25T11:20:55.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.344+0000 7fc31d330640 1 -- start start 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 0x7fc3181f08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3181592e0 0x7fc3180b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc3180c7340 con 0x7fc318153190 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc3180c7060 con 0x7fc3181592e0 2024-04-25T11:20:55.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc3180ca870 con 0x7fc3180c9570 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34918/0 (socket says 172.21.15.136:34918) 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 -- 172.21.15.136:0/3085830740 learned_addr learned my addr 172.21.15.136:0/3085830740 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc316ffd640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 0x7fc3181f08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 -- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 msgr2=0x7fc3181f08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3177fe640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3181592e0 0x7fc3180b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 0x7fc3181f08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 -- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3181592e0 msgr2=0x7fc3180b9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3181592e0 0x7fc3180b9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 -- 172.21.15.136:0/3085830740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3181f0de0 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc316ffd640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 0x7fc3181f08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc3167fc640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0b3040 tx=0x7fc30c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0b5070 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30c0b0c20 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc318157ca0 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0bb410 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc318158260 con 0x7fc318153190 2024-04-25T11:20:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc318157a10 con 0x7fc318153190 2024-04-25T11:20:55.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc30c0c2060 con 0x7fc318153190 2024-04-25T11:20:55.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 0x7fc304071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.348+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc30c12caf0 con 0x7fc318153190 2024-04-25T11:20:55.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.352+0000 7fc316ffd640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 0x7fc304071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.352+0000 7fc316ffd640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 0x7fc304071910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc308006fd0 tx=0x7fc308008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.352+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc30c018870 con 0x7fc318153190 2024-04-25T11:20:55.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.512+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc31806a350 con 0x7fc318153190 2024-04-25T11:20:55.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.516+0000 7fc2ff7fe640 1 -- 172.21.15.136:0/3085830740 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18550 (secure 0 0 0) 0x7fc30c0f98a0 con 0x7fc318153190 2024-04-25T11:20:55.514 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:55.514 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:24.451626+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 msgr2=0x7fc304071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 0x7fc304071910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc308006fd0 tx=0x7fc308008040 comp rx=0 tx=0).stop 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 msgr2=0x7fc318150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc30c0b3040 tx=0x7fc30c01bab0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 shutdown_connections 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3181592e0 0x7fc3180b9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc318153190 0x7fc318150140 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc30406f400 0x7fc304071910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 --2- 172.21.15.136:0/3085830740 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3180c9570 0x7fc3181f08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 >> 172.21.15.136:0/3085830740 conn(0x7fc3180c9d50 msgr2=0x7fc3181517e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 shutdown_connections 2024-04-25T11:20:55.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.520+0000 7fc31d330640 1 -- 172.21.15.136:0/3085830740 wait complete. 2024-04-25T11:20:55.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T11:20:55.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T11:20:55.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T11:20:55.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T11:20:55.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T11:20:55.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T11:20:55.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T11:20:55.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T11:20:55.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.756+0000 7f5471e15640 1 Processor -- start 2024-04-25T11:20:55.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.756+0000 7f5471e15640 1 -- start start 2024-04-25T11:20:55.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 Processor -- start 2024-04-25T11:20:55.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 -- start start 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f5471e15640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f546c1522f0 0x7f546c070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f5471e15640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f546c0c9780 0x7f546c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f5471e15640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f546c152700 con 0x7f546c0c9370 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f5471e15640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f546c0c8ee0 con 0x7f546c0c9780 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f5471e15640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f546c0c9b90 con 0x7f546c1522f0 2024-04-25T11:20:55.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f546c0c9780 0x7f546c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 Processor -- start 2024-04-25T11:20:55.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f546c0c9780 0x7f546c15bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34934/0 (socket says 172.21.15.136:34934) 2024-04-25T11:20:55.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 -- 172.21.15.136:0/516034591 learned_addr learned my addr 172.21.15.136:0/516034591 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 -- 172.21.15.136:0/516034591 >> v2:172.21.15.157:3300/0 conn(0x7f546c1522f0 msgr2=0x7f546c070700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:20:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 --2- 172.21.15.136:0/516034591 >> v2:172.21.15.157:3300/0 conn(0x7f546c1522f0 0x7f546c070700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 -- 172.21.15.136:0/516034591 >> v1:172.21.15.136:6789/0 conn(0x7f546c0c9370 legacy=0x7f546c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f546bfff640 1 -- 172.21.15.136:0/516034591 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c0729c0 con 0x7f546c0c9780 2024-04-25T11:20:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.756+0000 7fc24b28a640 1 Processor -- start 2024-04-25T11:20:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 -- start start 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc7680cbe10 0x7fc7680cc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7680ca5f0 con 0x7fc7681642a0 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7680cb950 con 0x7fc768168c10 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76d850640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc7680ca030 con 0x7fc7680cbe10 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1f400e28e0 0x7f1f400e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4004bcb0 con 0x7f1f400d9630 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4004b730 con 0x7f1f400e7c60 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f47fff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1f40047220 con 0x7f1f400e28e0 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f46ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1f400d9630 0x7f1f400060a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49622/0 (socket says 172.21.15.136:49622) 2024-04-25T11:20:55.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f46ffd640 1 -- 172.21.15.136:0/2183851160 learned_addr learned my addr 172.21.15.136:0/2183851160 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f467fc640 1 --2- 172.21.15.136:0/2183851160 >> v2:172.21.15.157:3300/0 conn(0x7f1f400e28e0 0x7f1f400e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24b28a640 1 -- start start 2024-04-25T11:20:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682817002 0 0) 0x7f1f4004bcb0 con 0x7f1f400d9630 2024-04-25T11:20:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f30003530 con 0x7f1f400d9630 2024-04-25T11:20:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24b28a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc244073420 0x7fc2441744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76c84e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc7681642a0 0x7fc768070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49638/0 (socket says 172.21.15.136:49638) 2024-04-25T11:20:55.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc76c84e640 1 -- 172.21.15.136:0/2247885220 learned_addr learned my addr 172.21.15.136:0/2247885220 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24b28a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc244163360 con 0x7fc24416cba0 2024-04-25T11:20:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24b28a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2440c8b90 con 0x7fc244162f50 2024-04-25T11:20:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24b28a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc244168d30 con 0x7fc244073420 2024-04-25T11:20:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249a87640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc24416cba0 0x7fc244072d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49628/0 (socket says 172.21.15.136:49628) 2024-04-25T11:20:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc767fff640 1 --2- 172.21.15.136:0/2247885220 >> v2:172.21.15.157:3300/0 conn(0x7fc7680cbe10 0x7fc7680cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068671327 0 0) 0x7f1f4004b730 con 0x7f1f400e7c60 2024-04-25T11:20:55.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f4004bcb0 con 0x7f1f400e7c60 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249a87640 1 -- 172.21.15.136:0/1974978337 learned_addr learned my addr 172.21.15.136:0/1974978337 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24aa89640 1 --2- 172.21.15.136:0/1974978337 >> v2:172.21.15.157:3300/0 conn(0x7fc244073420 0x7fc2441744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275266105 0 0) 0x7fc244163360 con 0x7fc24416cba0 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc230007940 con 0x7fc24416cba0 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596040798 0 0) 0x7fc2440c8b90 con 0x7fc244162f50 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc230007ca0 con 0x7fc244162f50 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24aa89640 1 -- 172.21.15.136:0/1974978337 >> v1:172.21.15.157:6789/0 conn(0x7fc244162f50 legacy=0x7fc2440c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24aa89640 1 -- 172.21.15.136:0/1974978337 >> v1:172.21.15.136:6789/0 conn(0x7fc24416cba0 legacy=0x7fc244072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7fc24aa89640 1 -- 172.21.15.136:0/1974978337 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc244169010 con 0x7fc244073420 2024-04-25T11:20:55.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.760+0000 7f1f467fc640 1 -- 172.21.15.136:0/2183851160 >> v1:172.21.15.157:6789/0 conn(0x7f1f400e7c60 legacy=0x7f1f400ea080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f1f467fc640 1 -- 172.21.15.136:0/2183851160 >> v1:172.21.15.136:6789/0 conn(0x7f1f400d9630 legacy=0x7f1f400060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f1f467fc640 1 -- 172.21.15.136:0/2183851160 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f400e3230 con 0x7f1f400e28e0 2024-04-25T11:20:55.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc767fff640 1 -- 172.21.15.136:0/2247885220 >> v1:172.21.15.157:6789/0 conn(0x7fc768168c10 legacy=0x7fc76816b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc7677fe640 1 -- 172.21.15.136:0/2247885220 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048537083 0 0) 0x7fc7680ca5f0 con 0x7fc7681642a0 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc767fff640 1 -- 172.21.15.136:0/2247885220 >> v1:172.21.15.136:6789/0 conn(0x7fc7681642a0 legacy=0x7fc768070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc767fff640 1 -- 172.21.15.136:0/2247885220 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7680ca310 con 0x7fc7680cbe10 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f546bfff640 1 --2- 172.21.15.136:0/516034591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9780 0x7f546c15bce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f546c071ee0 tx=0x7f546401ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bac3a4641b7d15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f546a7fc640 1 -- 172.21.15.136:0/516034591 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640b3070 con 0x7f546c0c9780 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f546a7fc640 1 -- 172.21.15.136:0/516034591 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54640ae400 con 0x7f546c0c9780 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f546a7fc640 1 -- 172.21.15.136:0/516034591 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640ae7d0 con 0x7f546c0c9780 2024-04-25T11:20:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fe7d8df2640 1 Processor -- start 2024-04-25T11:20:55.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f3496cef640 1 Processor -- start 2024-04-25T11:20:55.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 -- 172.21.15.136:0/516034591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9780 msgr2=0x7f546c15bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 --2- 172.21.15.136:0/516034591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9780 0x7f546c15bce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f546c071ee0 tx=0x7f546401ef50 comp rx=0 tx=0).stop 2024-04-25T11:20:55.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc24aa89640 1 --2- 172.21.15.136:0/1974978337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc244073420 0x7fc2441744d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00bd20 tx=0x7fc23c01f0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b767cdb1226d4f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 -- 172.21.15.136:0/516034591 shutdown_connections 2024-04-25T11:20:55.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 --2- 172.21.15.136:0/516034591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9780 0x7f546c15bce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 --2- 172.21.15.136:0/516034591 >> v2:172.21.15.157:3300/0 conn(0x7f546c1522f0 0x7f546c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 -- 172.21.15.136:0/516034591 >> 172.21.15.136:0/516034591 conn(0x7f546c0c9d50 msgr2=0x7f546c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f3496cef640 1 -- start start 2024-04-25T11:20:55.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 -- 172.21.15.136:0/516034591 shutdown_connections 2024-04-25T11:20:55.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0ae070 con 0x7fc244073420 2024-04-25T11:20:55.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f5471e15640 1 -- 172.21.15.136:0/516034591 wait complete. 2024-04-25T11:20:55.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c0b2400 con 0x7fc244073420 2024-04-25T11:20:55.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc249286640 1 -- 172.21.15.136:0/1974978337 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0b27b0 con 0x7fc244073420 2024-04-25T11:20:55.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fae54a92640 1 Processor -- start 2024-04-25T11:20:55.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fc767fff640 1 --2- 172.21.15.136:0/2247885220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7680cbe10 0x7fc7680cc220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc760009d90 tx=0x7fc76001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4424852462b134e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fae54a92640 1 -- start start 2024-04-25T11:20:55.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7fe7d8df2640 1 -- start start 2024-04-25T11:20:55.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.764+0000 7f1f467fc640 1 --2- 172.21.15.136:0/2183851160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400e28e0 0x7f1f400e2cf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f380097c0 tx=0x7f1f3801ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be83b76c78f219e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc7677fe640 1 -- 172.21.15.136:0/2247885220 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b4070 con 0x7fc7680cbe10 2024-04-25T11:20:55.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 -- 172.21.15.136:0/1974978337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc244073420 msgr2=0x7fc2441744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 --2- 172.21.15.136:0/1974978337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc244073420 0x7fc2441744d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00bd20 tx=0x7fc23c01f0a0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380b5070 con 0x7f1f400e28e0 2024-04-25T11:20:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc7677fe640 1 -- 172.21.15.136:0/2247885220 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600af600 con 0x7fc7680cbe10 2024-04-25T11:20:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc7677fe640 1 -- 172.21.15.136:0/2247885220 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600af9d0 con 0x7fc7680cbe10 2024-04-25T11:20:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 -- 172.21.15.136:0/1974978337 shutdown_connections 2024-04-25T11:20:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 --2- 172.21.15.136:0/1974978337 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc244073420 0x7fc2441744d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f380a8990 con 0x7f1f400e28e0 2024-04-25T11:20:55.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f45ffb640 1 -- 172.21.15.136:0/2183851160 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f380a8d40 con 0x7f1f400e28e0 2024-04-25T11:20:55.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 -- 172.21.15.136:0/1974978337 >> 172.21.15.136:0/1974978337 conn(0x7fc24406c840 msgr2=0x7fc24406cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 -- 172.21.15.136:0/1974978337 shutdown_connections 2024-04-25T11:20:55.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 -- 172.21.15.136:0/1974978337 wait complete. 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d8df2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 0x7fe7cc0ef460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d8df2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc0e7960 con 0x7fe7cc0dda20 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d8df2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc0e4620 con 0x7fe7cc0457e0 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d8df2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc0dde30 con 0x7fe7cc0ed040 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe7cc0dda20 0x7fe7cc0060a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49650/0 (socket says 172.21.15.136:49650) 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 0x7fe7cc0ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d37fe640 1 -- 172.21.15.136:0/137286875 learned_addr learned my addr 172.21.15.136:0/137286875 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:55.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3496cef640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34901522f0 0x7f3490070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3496cef640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f34900c9370 0x7f34900c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- 172.21.15.136:0/2247885220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7680cbe10 msgr2=0x7fc7680cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 --2- 172.21.15.136:0/2247885220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7680cbe10 0x7fc7680cc220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc760009d90 tx=0x7fc76001f150 comp rx=0 tx=0).stop 2024-04-25T11:20:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3496cef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3490152700 con 0x7f349015b5d0 2024-04-25T11:20:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- 172.21.15.136:0/2247885220 shutdown_connections 2024-04-25T11:20:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 --2- 172.21.15.136:0/2247885220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7680cbe10 0x7fc7680cc220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- 172.21.15.136:0/2247885220 >> 172.21.15.136:0/2247885220 conn(0x7fc768072650 msgr2=0x7fc768072a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 -- 172.21.15.136:0/2183851160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400e28e0 msgr2=0x7f1f400e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 --2- 172.21.15.136:0/2183851160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400e28e0 0x7f1f400e2cf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f380097c0 tx=0x7f1f3801ecb0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 -- 172.21.15.136:0/2183851160 shutdown_connections 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 --2- 172.21.15.136:0/2183851160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400e28e0 0x7f1f400e2cf0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f380097c0 tx=0x7f1f3801ecb0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 -- 172.21.15.136:0/2183851160 >> 172.21.15.136:0/2183851160 conn(0x7f1f400503c0 msgr2=0x7f1f400507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d3fff640 1 -- 172.21.15.136:0/137286875 >> v1:172.21.15.157:6789/0 conn(0x7fe7cc0457e0 legacy=0x7fe7cc04b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d3fff640 1 -- 172.21.15.136:0/137286875 >> v1:172.21.15.136:6789/0 conn(0x7fe7cc0dda20 legacy=0x7fe7cc0060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fe7d3fff640 1 -- 172.21.15.136:0/137286875 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc0e3860 con 0x7fe7cc0ed040 2024-04-25T11:20:55.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- 172.21.15.136:0/2247885220 shutdown_connections 2024-04-25T11:20:55.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3496cef640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f34900c8ee0 con 0x7f34900c9370 2024-04-25T11:20:55.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3496cef640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f34900729c0 con 0x7f34901522f0 2024-04-25T11:20:55.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- 172.21.15.136:0/2247885220 wait complete. 2024-04-25T11:20:55.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f349528c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f349015b5d0 0x7f349015b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49666/0 (socket says 172.21.15.136:49666) 2024-04-25T11:20:55.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f349528c640 1 -- 172.21.15.136:0/811740941 learned_addr learned my addr 172.21.15.136:0/811740941 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:55.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 -- 172.21.15.136:0/2183851160 shutdown_connections 2024-04-25T11:20:55.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f1f47fff640 1 -- 172.21.15.136:0/2183851160 wait complete. 2024-04-25T11:20:55.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355671815 0 0) 0x7f3490152700 con 0x7f349015b5d0 2024-04-25T11:20:55.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3478003530 con 0x7f349015b5d0 2024-04-25T11:20:55.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348ffff640 1 --2- 172.21.15.136:0/811740941 >> v2:172.21.15.136:3300/0 conn(0x7f34900c9370 0x7f34900c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc24b28a640 1 Processor -- start 2024-04-25T11:20:55.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f3494a8b640 1 --2- 172.21.15.136:0/811740941 >> v2:172.21.15.157:3300/0 conn(0x7f34901522f0 0x7f3490070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348ffff640 1 -- 172.21.15.136:0/811740941 >> v2:172.21.15.157:3300/0 conn(0x7f34901522f0 msgr2=0x7f3490070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348ffff640 1 --2- 172.21.15.136:0/811740941 >> v2:172.21.15.157:3300/0 conn(0x7f34901522f0 0x7f3490070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348ffff640 1 -- 172.21.15.136:0/811740941 >> v1:172.21.15.136:6789/0 conn(0x7f349015b5d0 legacy=0x7f349015b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1813083014 0 0) 0x7f3478003530 con 0x7f349015b5d0 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7f348ffff640 1 -- 172.21.15.136:0/811740941 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3490073f30 con 0x7f34900c9370 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 Processor -- start 2024-04-25T11:20:55.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.768+0000 7fc76d850640 1 -- start start 2024-04-25T11:20:55.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f47fff640 1 Processor -- start 2024-04-25T11:20:55.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d3fff640 1 --2- 172.21.15.136:0/137286875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 0x7fe7cc0ef460 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80097f0 tx=0x7fe7c801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3131a4609fb35b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f47fff640 1 -- start start 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 0x7fae500cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 0x7fae5015c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fae5015ca40 0x7fae500d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fae500ce7c0 con 0x7fae5015ca40 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fae5015f820 con 0x7fae500c6460 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae54a92640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fae5015fb00 con 0x7fae500ce3b0 2024-04-25T11:20:55.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d27fc640 1 -- 172.21.15.136:0/137286875 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80b0070 con 0x7fe7cc0ed040 2024-04-25T11:20:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d27fc640 1 -- 172.21.15.136:0/137286875 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c80ab600 con 0x7fe7cc0ed040 2024-04-25T11:20:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d27fc640 1 -- 172.21.15.136:0/137286875 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80ab9d0 con 0x7fe7cc0ed040 2024-04-25T11:20:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fae5015ca40 0x7fae500d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fae5015ca40 0x7fae500d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34958/0 (socket says 172.21.15.136:34958) 2024-04-25T11:20:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f348ffff640 1 --2- 172.21.15.136:0/811740941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34900c9370 0x7f34900c9780 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3490071ee0 tx=0x7f348001c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec79a1d398fbc656 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 -- 172.21.15.136:0/2023679581 learned_addr learned my addr 172.21.15.136:0/2023679581 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7680cbe10 0x7fc7680c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 -- start start 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc244073420 0x7fc24416aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc244162f50 0x7fc244202b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc24416cba0 0x7fc2442030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441698b0 con 0x7fc244073420 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc2441695d0 con 0x7fc24416cba0 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24b28a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc24416cfb0 con 0x7fc244162f50 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7681642a0 0x7fc7680cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800b0070 con 0x7f34900c9370 2024-04-25T11:20:55.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc768168c10 0x7fc768145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34800ab400 con 0x7f34900c9370 2024-04-25T11:20:55.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc249a87640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc244162f50 0x7fc244202b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7681646b0 con 0x7fc768168c10 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc768070750 con 0x7fc7681642a0 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d850640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc76806df90 con 0x7fc7680cbe10 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76c84e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7680cbe10 0x7fc7680c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76c84e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7680cbe10 0x7fc7680c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45094/0 (socket says 172.21.15.136:45094) 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76c84e640 1 -- 172.21.15.136:0/754569631 learned_addr learned my addr 172.21.15.136:0/754569631 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7681642a0 0x7fc7680cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc24416cba0 0x7fc2442030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc249a87640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc244162f50 0x7fc244202b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45078/0 (socket says 172.21.15.136:45078) 2024-04-25T11:20:55.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc249a87640 1 -- 172.21.15.136:0/265136883 learned_addr learned my addr 172.21.15.136:0/265136883 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4f7fe640 1 --2- 172.21.15.136:0/2023679581 >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 0x7fae500cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24a288640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc244073420 0x7fc24416aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d04f640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc768168c10 0x7fc768145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 -- 172.21.15.136:0/2023679581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 msgr2=0x7fae5015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 -- 172.21.15.136:0/137286875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 msgr2=0x7fe7cc0ef460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 --2- 172.21.15.136:0/137286875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 0x7fe7cc0ef460 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80097f0 tx=0x7fe7c801c4e0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae46ffd640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 0x7fae5015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400d9630 0x7f1f40176130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f400e7c60 0x7f1f400e1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 -- 172.21.15.136:0/137286875 shutdown_connections 2024-04-25T11:20:55.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 --2- 172.21.15.136:0/137286875 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0ed040 0x7fe7cc0ef460 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 -- 172.21.15.136:0/137286875 >> 172.21.15.136:0/137286875 conn(0x7fe7cc0503c0 msgr2=0x7fe7cc0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f40179480 0x7f1f40179890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 -- 172.21.15.136:0/811740941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34900c9370 msgr2=0x7f34900c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f467fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f400e7c60 0x7f1f400e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 -- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc244162f50 msgr2=0x7fc244202b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc244162f50 0x7fc244202b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 -- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc244073420 msgr2=0x7fc24416aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc244073420 0x7fc24416aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc24aa89640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24416afc0 con 0x7fc24416cba0 2024-04-25T11:20:55.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f348f7fe640 1 -- 172.21.15.136:0/811740941 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800b0070 con 0x7f34900c9370 2024-04-25T11:20:55.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 -- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7680cbe10 msgr2=0x7fc7680c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 --2- 172.21.15.136:0/811740941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34900c9370 0x7f34900c9780 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3490071ee0 tx=0x7f348001c310 comp rx=0 tx=0).stop 2024-04-25T11:20:55.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7680cbe10 0x7fc7680c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 -- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc768168c10 msgr2=0x7fc768145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f1f467fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f400e7c60 0x7f1f400e1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45120/0 (socket says 172.21.15.136:45120) 2024-04-25T11:20:55.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 0x7fae5015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc768168c10 0x7fc768145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc767fff640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76815a030 con 0x7fc7681642a0 2024-04-25T11:20:55.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 -- 172.21.15.136:0/811740941 shutdown_connections 2024-04-25T11:20:55.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 -- 172.21.15.136:0/2023679581 >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 msgr2=0x7fae500cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 --2- 172.21.15.136:0/811740941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34900c9370 0x7f34900c9780 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3490071ee0 tx=0x7f348001c310 comp rx=0 tx=0).stop 2024-04-25T11:20:55.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 --2- 172.21.15.136:0/811740941 >> v2:172.21.15.157:3300/0 conn(0x7f34901522f0 0x7f3490070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 --2- 172.21.15.136:0/2023679581 >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 0x7fae500cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fc76d04f640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc768168c10 0x7fc768145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:55.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7f3496cef640 1 -- 172.21.15.136:0/811740941 >> 172.21.15.136:0/811740941 conn(0x7f34900c9d50 msgr2=0x7f34900ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4ffff640 1 -- 172.21.15.136:0/2023679581 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5015fde0 con 0x7fae5015ca40 2024-04-25T11:20:55.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae46ffd640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 0x7fae5015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:55.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fae4f7fe640 1 --2- 172.21.15.136:0/2023679581 >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 0x7fae500cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:55.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.772+0000 7fe7d8df2640 1 -- 172.21.15.136:0/137286875 shutdown_connections 2024-04-25T11:20:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f3496cef640 1 -- 172.21.15.136:0/811740941 shutdown_connections 2024-04-25T11:20:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fe7d8df2640 1 -- 172.21.15.136:0/137286875 wait complete. 2024-04-25T11:20:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f3496cef640 1 -- 172.21.15.136:0/811740941 wait complete. 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f46ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400d9630 0x7f1f40176130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f46ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400d9630 0x7f1f40176130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41022/0 (socket says 172.21.15.136:41022) 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f467fc640 1 -- 172.21.15.136:0/3960314459 learned_addr learned my addr 172.21.15.136:0/3960314459 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4004b9d0 con 0x7f1f40179480 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1f400d9a40 con 0x7f1f400d9630 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1f40046d00 con 0x7f1f400e7c60 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f40179480 0x7f1f40179890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 -- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f400e7c60 msgr2=0x7f1f400e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f400e7c60 0x7f1f400e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 -- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400d9630 msgr2=0x7f1f40176130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f400d9630 0x7f1f40176130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f40176670 con 0x7f1f40179480 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f477fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f40179480 0x7f1f40179890 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1f480520f0 tx=0x7f1f4806aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f5471e15640 1 Processor -- start 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f5471e15640 1 -- start start 2024-04-25T11:20:55.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48091070 con 0x7f1f40179480 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4806dbd0 con 0x7f1f40179480 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f480953b0 con 0x7f1f40179480 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc767fff640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7681642a0 0x7fc7680cdb40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7680732b0 tx=0x7fc76001bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600c0020 con 0x7fc7681642a0 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7680c9070 con 0x7fc7681642a0 2024-04-25T11:20:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7680cfed0 con 0x7fc7681642a0 2024-04-25T11:20:55.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600ba470 con 0x7fc7681642a0 2024-04-25T11:20:55.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b4040 con 0x7fc7681642a0 2024-04-25T11:20:55.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f400e0dc0 con 0x7f1f40179480 2024-04-25T11:20:55.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f400d2bc0 con 0x7f1f40179480 2024-04-25T11:20:55.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc74effd640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7680cf9a0 con 0x7fc7681642a0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc24aa89640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc24416cba0 0x7fc2442030c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2440714a0 tx=0x7fc23c00a2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0ae040 con 0x7fc24416cba0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc24b28a640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2440cbdc0 con 0x7fc24416cba0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c0b2880 con 0x7fc24416cba0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc24b28a640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24415e9a0 con 0x7fc24416cba0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc24b28a640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc24406a2e0 con 0x7fc24416cba0 2024-04-25T11:20:55.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0b2c30 con 0x7fc24416cba0 2024-04-25T11:20:55.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.776+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1f40005580 con 0x7f1f40179480 2024-04-25T11:20:55.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 Processor -- start 2024-04-25T11:20:55.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc7600c3070 con 0x7fc7681642a0 2024-04-25T11:20:55.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc75406f430 0x7fc754071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc76012d320 con 0x7fc7681642a0 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7fc754078b70 0x7fc75407b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7600af600 con 0x7fc754078b70 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc23c0c3020 con 0x7fc24416cba0 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc22806f400 0x7fc228071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc23c12dd80 con 0x7fc24416cba0 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc228078b40 0x7fc22807afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc22807b6a0 con 0x7fc228078b40 2024-04-25T11:20:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc23c12e170 con 0x7fc24416cba0 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9370 0x7f546c1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f546c0c9780 0x7f546c148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 0x7f546c1469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f546c15c3a0 con 0x7f546c0c9370 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f546c15c220 con 0x7f546c1522f0 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f546c165180 con 0x7f546c0c9780 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 0x7f546c1469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f546c0c9780 0x7f546c148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 0x7f546c1469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41034/0 (socket says 172.21.15.136:41034) 2024-04-25T11:20:55.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 -- 172.21.15.136:0/1375961231 learned_addr learned my addr 172.21.15.136:0/1375961231 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc76c84e640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc75406f430 0x7fc754071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc249a87640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc228078b40 0x7fc22807afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc24a288640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc22806f400 0x7fc228071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 Processor -- start 2024-04-25T11:20:55.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 -- start start 2024-04-25T11:20:55.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc76d04f640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7fc754078b70 0x7fc75407b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546b7fe640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9370 0x7f546c1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fae4ffff640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae500d13d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fae5006dd60 tx=0x7fae4801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd28135c2f96065 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc249a87640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc228078b40 0x7fc22807afd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 -- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f546c0c9780 msgr2=0x7f546c148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f546c0c9780 0x7f546c148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 -- 172.21.15.136:0/1375961231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9370 msgr2=0x7f546c1f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9370 0x7f546c1f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546c1f23e0 con 0x7f546c1522f0 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546b7fe640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f546c0c9370 0x7f546c1f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 -- start start 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7cc0457e0 0x7fe7cc0ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 0x7fe7cc0e2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7cc0ed040 0x7fe7cc181440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0e7670 con 0x7fe7cc0457e0 2024-04-25T11:20:55.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0e7c20 con 0x7fe7cc0ed040 2024-04-25T11:20:55.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d8df2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc0067b0 con 0x7fe7cc0dda20 2024-04-25T11:20:55.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f546bfff640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 0x7f546c1469b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f546401cb80 tx=0x7f546401bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1f48094070 con 0x7f1f40179480 2024-04-25T11:20:55.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc76c84e640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc75406f430 0x7fc754071940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000f70 tx=0x7fc75c007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc76d04f640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7fc754078b70 0x7fc75407b000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc24a288640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc22806f400 0x7fc228071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2340011f0 tx=0x7fc234009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== osd.2 v2:172.21.15.136:6816/4197312530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc22807b6a0 con 0x7fc228078b40 2024-04-25T11:20:55.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc760134050 con 0x7fc7681642a0 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 0x7fe7cc0e2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 0x7fe7cc0e2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45136/0 (socket says 172.21.15.136:45136) 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d2ffd640 1 -- 172.21.15.136:0/1801845613 learned_addr learned my addr 172.21.15.136:0/1801845613 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fe7d3fff640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7cc0ed040 0x7fe7cc181440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== osd.1 v2:172.21.15.136:6824/1933356212 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7600af600 con 0x7fc754078b70 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f1f2f7fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f3406f450 0x7f1f34071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f1f46ffd640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f3406f450 0x7f1f34071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fae4effd640 1 -- 172.21.15.136:0/2023679581 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480ac070 con 0x7fae5015ca40 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fae4effd640 1 -- 172.21.15.136:0/2023679581 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae480a8420 con 0x7fae5015ca40 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7fae4effd640 1 -- 172.21.15.136:0/2023679581 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480a87f0 con 0x7fae5015ca40 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1f481090b0 con 0x7f1f40179480 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f2f7fe640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7f1f34078c10 0x7f1f3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f3407b770 con 0x7f1f34078c10 2024-04-25T11:20:55.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1f48110060 con 0x7f1f40179480 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 -- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7cc0ed040 msgr2=0x7fe7cc181440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7cc0ed040 0x7fe7cc181440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 -- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7cc0457e0 msgr2=0x7fe7cc0ef1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7cc0457e0 0x7fe7cc0ef1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc0e3630 con 0x7fe7cc0dda20 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 0x7f34901583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f349015b5d0 0x7f3490148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3490155cc0 0x7f3490152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3490165510 con 0x7f3490155cc0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3490165390 con 0x7f349015b5d0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f3496cef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3490165690 con 0x7f34901522f0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 0x7f34901583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 0x7f34901583d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45152/0 (socket says 172.21.15.136:45152) 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 -- 172.21.15.136:0/1470446370 learned_addr learned my addr 172.21.15.136:0/1470446370 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f348ffff640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f349015b5d0 0x7f3490148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640b3040 con 0x7f546c1522f0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546c0ccab0 con 0x7f546c1522f0 2024-04-25T11:20:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.780+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546c1f16c0 con 0x7f546c1522f0 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546400e430 con 0x7f546c1522f0 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546400e800 con 0x7f546c1522f0 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f546c155e50 con 0x7f546c1522f0 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- 172.21.15.136:0/2023679581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 msgr2=0x7fae500d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae500d13d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fae5006dd60 tx=0x7fae4801c730 comp rx=0 tx=0).stop 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- 172.21.15.136:0/2023679581 shutdown_connections 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae500d13d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 --2- 172.21.15.136:0/2023679581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500ce3b0 0x7fae5015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 --2- 172.21.15.136:0/2023679581 >> v2:172.21.15.157:3300/0 conn(0x7fae500c6460 0x7fae500cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- 172.21.15.136:0/2023679581 >> 172.21.15.136:0/2023679581 conn(0x7fae5006d700 msgr2=0x7fae5006e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 -- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f349015b5d0 msgr2=0x7f3490148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f349015b5d0 0x7f3490148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 -- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3490155cc0 msgr2=0x7f3490152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f349528c640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3490155cc0 0x7f3490152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3490155cc0 0x7f3490152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3490074c00 con 0x7f34901522f0 2024-04-25T11:20:55.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- 172.21.15.136:0/2023679581 shutdown_connections 2024-04-25T11:20:55.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f467fc640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7f1f34078c10 0x7f1f3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d2ffd640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 0x7fe7cc0e2700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c3a0 tx=0x7fe7c000c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f46ffd640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f3406f450 0x7f1f34071960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c007fd0 tx=0x7f1f3c007d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f3494a8b640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 0x7f34901583d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3484009f20 tx=0x7f348400d4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00052d0 con 0x7fe7cc0dda20 2024-04-25T11:20:55.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc0ef700 con 0x7fe7cc0dda20 2024-04-25T11:20:55.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc035b00 con 0x7fe7cc0dda20 2024-04-25T11:20:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f1f467fc640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7f1f34078c10 0x7f1f3407b0a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348400b250 con 0x7f34901522f0 2024-04-25T11:20:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c0005470 con 0x7fe7cc0dda20 2024-04-25T11:20:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0007260 con 0x7fe7cc0dda20 2024-04-25T11:20:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe7b4001040 con 0x7fe7cc0dda20 2024-04-25T11:20:55.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348400b3f0 con 0x7f34901522f0 2024-04-25T11:20:55.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348400a660 con 0x7f34901522f0 2024-04-25T11:20:55.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7f349528c640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3490155cc0 0x7f3490152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:55.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- 172.21.15.136:0/2023679581 wait complete. 2024-04-25T11:20:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 Processor -- start 2024-04-25T11:20:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3490158910 con 0x7f34901522f0 2024-04-25T11:20:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34901fdfe0 con 0x7f34901522f0 2024-04-25T11:20:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f349006a350 con 0x7f34901522f0 2024-04-25T11:20:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== osd.4 v2:172.21.15.157:6808/3530978019 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3407b770 con 0x7f1f34078c10 2024-04-25T11:20:55.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.784+0000 7fae54a92640 1 -- start start 2024-04-25T11:20:55.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 Processor -- start 2024-04-25T11:20:55.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 -- start start 2024-04-25T11:20:55.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe7c0007400 con 0x7fe7cc0dda20 2024-04-25T11:20:55.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7b806f430 0x7fe7b8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f54640c1030 con 0x7f546c1522f0 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5458077c10 0x7f545807a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f546412af00 con 0x7f546c1522f0 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7f5458081350 0x7f54580837e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5458083eb0 con 0x7f5458081350 2024-04-25T11:20:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f546412b2b0 con 0x7f546c1522f0 2024-04-25T11:20:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe7c008f1d0 con 0x7fe7cc0dda20 2024-04-25T11:20:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fe7b8078bf0 0x7fe7b807b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f546affd640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7f5458081350 0x7f54580837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f546b7fe640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5458077c10 0x7f545807a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7c0014e50 con 0x7fe7b8078bf0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe7c008f5c0 con 0x7fe7cc0dda20 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7d3fff640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fe7b8078bf0 0x7fe7b807b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fe7d37fe640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7b806f430 0x7fe7b8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f348400a800 con 0x7f34901522f0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f546affd640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7f5458081350 0x7f54580837e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7f546b7fe640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5458077c10 0x7f545807a120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f545c009890 tx=0x7f545c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 0x7fd4b40352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4b4036c50 0x7fd4b4037060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4008780 0x7fd4b4008b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd4b40dd470 con 0x7fd4b4034ed0 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd4b40dd710 con 0x7fd4b4036c50 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4c1539640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b40dd9d0 con 0x7fd4b4008780 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4b4036c50 0x7fd4b4037060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4b4036c50 0x7fd4b4037060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41058/0 (socket says 172.21.15.136:41058) 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 -- 172.21.15.136:0/351068156 learned_addr learned my addr 172.21.15.136:0/351068156 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bbfff640 1 --2- 172.21.15.136:0/351068156 >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 0x7fd4b40352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 -- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4008780 msgr2=0x7fd4b4008b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 --2- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4008780 0x7fd4b4008b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 -- 172.21.15.136:0/351068156 >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 msgr2=0x7fd4b40352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.788+0000 7fd4bb7fe640 1 --2- 172.21.15.136:0/351068156 >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 0x7fd4b40352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fd4bb7fe640 1 -- 172.21.15.136:0/351068156 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b40ddcb0 con 0x7fd4b4036c50 2024-04-25T11:20:55.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fd4bbfff640 1 --2- 172.21.15.136:0/351068156 >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 0x7fd4b40352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500c6460 0x7fae502061b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae500ce3b0 0x7fae50157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae5015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f347c06f4d0 0x7f347c0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3484090b50 con 0x7f34901522f0 2024-04-25T11:20:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fe7d37fe640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7b806f430 0x7fe7b8071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe7d406b910 tx=0x7fe7d4069810 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f347c078c90 0x7f347c07b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348ffff640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f347c06f4d0 0x7f347c0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3484019ec0 con 0x7f347c078c90 2024-04-25T11:20:55.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3484090f40 con 0x7f34901522f0 2024-04-25T11:20:55.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae5015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae5015acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35030/0 (socket says 172.21.15.136:35030) 2024-04-25T11:20:55.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 -- 172.21.15.136:0/682996041 learned_addr learned my addr 172.21.15.136:0/682996041 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae46ffd640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae500ce3b0 0x7fae50157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f349528c640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f347c078c90 0x7f347c07b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fe7d3fff640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fe7b8078bf0 0x7fe7b807b080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f349528c640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f347c078c90 0x7f347c07b120 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fae500c6870 con 0x7fae5015ca40 2024-04-25T11:20:55.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae500c85e0 con 0x7fae500ce3b0 2024-04-25T11:20:55.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae54a92640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fae5015f4c0 con 0x7fae500c6460 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 -- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500c6460 msgr2=0x7fae502061b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae500c6460 0x7fae502061b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 -- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae500ce3b0 msgr2=0x7fae50157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae500ce3b0 0x7fae50157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae50205710 con 0x7fae5015ca40 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4ffff640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae5015acb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae4801ade0 tx=0x7fae480194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348ffff640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f347c06f4d0 0x7f347c0719e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34800ad000 tx=0x7f34800a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== osd.0 v2:172.21.15.136:6800/3420887400 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7c0014e50 con 0x7fe7b8078bf0 2024-04-25T11:20:55.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== osd.3 v2:172.21.15.136:6808/2427382821 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3484019ec0 con 0x7f347c078c90 2024-04-25T11:20:55.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== osd.5 v2:172.21.15.157:6800/4215140907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5458083eb0 con 0x7f5458081350 2024-04-25T11:20:55.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480bf070 con 0x7fae5015ca40 2024-04-25T11:20:55.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae480090d0 con 0x7fae5015ca40 2024-04-25T11:20:55.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae480ac040 con 0x7fae5015ca40 2024-04-25T11:20:55.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fd4bb7fe640 1 --2- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4036c50 0x7fd4b4037060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4051430 tx=0x7fd4a8024c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c975e0998f51f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fd4baffd640 1 -- 172.21.15.136:0/351068156 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a80adac0 con 0x7fd4b4036c50 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.792+0000 7fd4baffd640 1 -- 172.21.15.136:0/351068156 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a80adc60 con 0x7fd4b4036c50 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 -- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4036c50 msgr2=0x7fd4b4037060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 --2- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4036c50 0x7fd4b4037060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4051430 tx=0x7fd4a8024c30 comp rx=0 tx=0).stop 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 -- 172.21.15.136:0/351068156 shutdown_connections 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 --2- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4008780 0x7fd4b4008b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 --2- 172.21.15.136:0/351068156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4036c50 0x7fd4b4037060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 --2- 172.21.15.136:0/351068156 >> v2:172.21.15.136:3300/0 conn(0x7fd4b4034ed0 0x7fd4b40352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 -- 172.21.15.136:0/351068156 >> 172.21.15.136:0/351068156 conn(0x7fd4b40514d0 msgr2=0x7fd4b40518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 -- 172.21.15.136:0/351068156 shutdown_connections 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.796+0000 7fd4c1539640 1 -- 172.21.15.136:0/351068156 wait complete. 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.800+0000 7fd4c1539640 1 Processor -- start 2024-04-25T11:20:55.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.800+0000 7fd4c1539640 1 -- start start 2024-04-25T11:20:55.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.800+0000 7fae54a92640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae50156bb0 con 0x7fae5015ca40 2024-04-25T11:20:55.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.800+0000 7fae54a92640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae502059b0 con 0x7fae5015ca40 2024-04-25T11:20:55.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.800+0000 7fd4c1539640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4008780 0x7fd4b404f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae5006a2e0 con 0x7fae5015ca40 2024-04-25T11:20:55.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c1539640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4034ed0 0x7fd4b40d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c1539640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 0x7fd4b40cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc24b28a640 1 -- 172.21.15.136:0/265136883 --> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc24406a2e0 con 0x7fc228078b40 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4bbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4008780 0x7fd4b404f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4bb7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4034ed0 0x7fd4b40d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4bbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4008780 0x7fd4b404f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41064/0 (socket says 172.21.15.136:41064) 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4bb7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4034ed0 0x7fd4b40d2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45174/0 (socket says 172.21.15.136:45174) 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c0d38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 0x7fd4b40cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c0d38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 0x7fd4b40cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35046/0 (socket says 172.21.15.136:35046) 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc23affd640 1 -- 172.21.15.136:0/265136883 <== osd.2 v2:172.21.15.136:6816/4197312530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc24406a2e0 con 0x7fc228078b40 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc228078b40 msgr2=0x7fc22807afd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc228078b40 0x7fc22807afd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc22806f400 msgr2=0x7fc228071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc22806f400 0x7fc228071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2340011f0 tx=0x7fc234009040 comp rx=0 tx=0).stop 2024-04-25T11:20:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc24416cba0 msgr2=0x7fc2442030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 --2- 172.21.15.136:0/265136883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc24416cba0 0x7fc2442030c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc2440714a0 tx=0x7fc23c00a2f0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc24aa89640 1 -- 172.21.15.136:0/265136883 reap_dead start 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 shutdown_connections 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 >> 172.21.15.136:0/265136883 conn(0x7fc24406c840 msgr2=0x7fc244070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4bbfff640 1 -- 172.21.15.136:0/652678266 learned_addr learned my addr 172.21.15.136:0/652678266 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:55.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 shutdown_connections 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fc238ff9640 1 -- 172.21.15.136:0/265136883 wait complete. 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fae480c2070 con 0x7fae5015ca40 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae3006f450 0x7fae30071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fae48129df0 con 0x7fae5015ca40 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7fae30078c10 0x7fae3007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae480020c0 con 0x7fae30078c10 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fae4812a2d0 con 0x7fae5015ca40 2024-04-25T11:20:55.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b40dd160 con 0x7fd4b4036c50 2024-04-25T11:20:55.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae4f7fe640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae3006f450 0x7fae30071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fae46ffd640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7fae30078c10 0x7fae3007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b4048480 con 0x7fd4b4008780 2024-04-25T11:20:55.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.804+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b40d7650 con 0x7fd4b4034ed0 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fae46ffd640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7fae30078c10 0x7fae3007b0a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 -- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4034ed0 msgr2=0x7fd4b40d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4b4034ed0 0x7fd4b40d2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 -- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4008780 msgr2=0x7fd4b404f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4b4008780 0x7fd4b404f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b40dbb50 con 0x7fd4b4036c50 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c0d38640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 0x7fd4b40cfdc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0520f0 tx=0x7fd4bc06aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc091070 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4bc06ddd0 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b40d3520 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc0955b0 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b404ff00 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd4b40501e0 con 0x7fd4b4036c50 2024-04-25T11:20:55.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.808+0000 7fae4f7fe640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae3006f450 0x7fae30071960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae400098f0 tx=0x7fae40009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd4bc094070 con 0x7fd4b4036c50 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd4a006f430 0x7fd4a0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd4bc10bcc0 con 0x7fd4b4036c50 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7fd4a0078b70 0x7fd4a007b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4a007b6d0 con 0x7fd4a0078b70 2024-04-25T11:20:55.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd4bc10e390 con 0x7fd4b4036c50 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4bbfff640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd4a006f430 0x7fd4a0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4bb7fe640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7fd4a0078b70 0x7fd4a007b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4bbfff640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd4a006f430 0x7fd4a0071940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000abb0 tx=0x7fd4b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== osd.6 v2:172.21.15.157:6824/224466234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae480020c0 con 0x7fae30078c10 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 --> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f349006a350 con 0x7f347c078c90 2024-04-25T11:20:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7fd4bb7fe640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7fd4a0078b70 0x7fd4a007b000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:55.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f348d7fa640 1 -- 172.21.15.136:0/1470446370 <== osd.3 v2:172.21.15.136:6808/2427382821 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f349006a350 con 0x7f347c078c90 2024-04-25T11:20:55.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f347c078c90 msgr2=0x7f347c07b120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f347c078c90 0x7f347c07b120 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f347c06f4d0 msgr2=0x7f347c0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f347c06f4d0 0x7f347c0719e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34800ad000 tx=0x7f34800a6000 comp rx=0 tx=0).stop 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 msgr2=0x7f34901583d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 --2- 172.21.15.136:0/1470446370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34901522f0 0x7f34901583d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3484009f20 tx=0x7f348400d4c0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f349528c640 1 -- 172.21.15.136:0/1470446370 reap_dead start 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 shutdown_connections 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 >> 172.21.15.136:0/1470446370 conn(0x7f34900c9d50 msgr2=0x7f34900c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.812+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 --> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1f400e1e80 con 0x7f1f34078c10 2024-04-25T11:20:55.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 shutdown_connections 2024-04-25T11:20:55.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f3496cef640 1 -- 172.21.15.136:0/1470446370 wait complete. 2024-04-25T11:20:55.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f2f7fe640 1 -- 172.21.15.136:0/3960314459 <== osd.4 v2:172.21.15.157:6808/3530978019 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1f400e1e80 con 0x7f1f34078c10 2024-04-25T11:20:55.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== osd.7 v2:172.21.15.157:6809/498117293 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd4a007b6d0 con 0x7fd4a0078b70 2024-04-25T11:20:55.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7fc74effd640 1 -- 172.21.15.136:0/754569631 --> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc7680c6c30 con 0x7fc754078b70 2024-04-25T11:20:55.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7f1f34078c10 msgr2=0x7f1f3407b0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7fc7657fa640 1 -- 172.21.15.136:0/754569631 <== osd.1 v2:172.21.15.136:6824/1933356212 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc7680c6c30 con 0x7fc754078b70 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7f1f34078c10 0x7f1f3407b0a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f3406f450 msgr2=0x7f1f34071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f3406f450 0x7f1f34071960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c007fd0 tx=0x7f1f3c007d80 comp rx=0 tx=0).stop 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f40179480 msgr2=0x7f1f40179890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 --2- 172.21.15.136:0/3960314459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f40179480 0x7f1f40179890 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1f480520f0 tx=0x7f1f4806aca0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f477fe640 1 -- 172.21.15.136:0/3960314459 reap_dead start 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 shutdown_connections 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 >> 172.21.15.136:0/3960314459 conn(0x7f1f400503c0 msgr2=0x7f1f400c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 shutdown_connections 2024-04-25T11:20:55.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.816+0000 7f1f47fff640 1 -- 172.21.15.136:0/3960314459 wait complete. 2024-04-25T11:20:55.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 --> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f546c155e50 con 0x7f5458081350 2024-04-25T11:20:55.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 --> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe7b4002de0 con 0x7fe7b8078bf0 2024-04-25T11:20:55.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5453fff640 1 -- 172.21.15.136:0/1375961231 <== osd.5 v2:172.21.15.157:6800/4215140907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f546c155e50 con 0x7f5458081350 2024-04-25T11:20:55.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7bffff640 1 -- 172.21.15.136:0/1801845613 <== osd.0 v2:172.21.15.136:6800/3420887400 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe7b4002de0 con 0x7fe7b8078bf0 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7f5458081350 msgr2=0x7f54580837e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7f5458081350 0x7f54580837e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5458077c10 msgr2=0x7f545807a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5458077c10 0x7f545807a120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f545c009890 tx=0x7f545c008040 comp rx=0 tx=0).stop 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 msgr2=0x7f546c1469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 --2- 172.21.15.136:0/1375961231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f546c1522f0 0x7f546c1469b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f546401cb80 tx=0x7f546401bcf0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f546bfff640 1 -- 172.21.15.136:0/1375961231 reap_dead start 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 shutdown_connections 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 >> 172.21.15.136:0/1375961231 conn(0x7f546c0c9d50 msgr2=0x7f546c06e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 shutdown_connections 2024-04-25T11:20:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7f5471e15640 1 -- 172.21.15.136:0/1375961231 wait complete. 2024-04-25T11:20:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fe7b8078bf0 msgr2=0x7fe7b807b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fe7b8078bf0 0x7fe7b807b080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7b806f430 msgr2=0x7fe7b8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7b806f430 0x7fe7b8071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe7d406b910 tx=0x7fe7d4069810 comp rx=0 tx=0).stop 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 msgr2=0x7fe7cc0e2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 --2- 172.21.15.136:0/1801845613 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7cc0dda20 0x7fe7cc0e2700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c3a0 tx=0x7fe7c000c860 comp rx=0 tx=0).stop 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d3fff640 1 -- 172.21.15.136:0/1801845613 reap_dead start 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 shutdown_connections 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 >> 172.21.15.136:0/1801845613 conn(0x7fe7cc0503c0 msgr2=0x7fe7cc0d7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 shutdown_connections 2024-04-25T11:20:55.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.824+0000 7fe7d8df2640 1 -- 172.21.15.136:0/1801845613 wait complete. 2024-04-25T11:20:55.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7fc754078b70 msgr2=0x7fc75407b000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7fc754078b70 0x7fc75407b000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc75406f430 msgr2=0x7fc754071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc75406f430 0x7fc754071940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000f70 tx=0x7fc75c007ee0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7681642a0 msgr2=0x7fc7680cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 --2- 172.21.15.136:0/754569631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7681642a0 0x7fc7680cdb40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc7680732b0 tx=0x7fc76001bf10 comp rx=0 tx=0).stop 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d04f640 1 -- 172.21.15.136:0/754569631 reap_dead start 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 shutdown_connections 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 >> 172.21.15.136:0/754569631 conn(0x7fc768072650 msgr2=0x7fc76806e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 shutdown_connections 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.828+0000 7fc76d850640 1 -- 172.21.15.136:0/754569631 wait complete. 2024-04-25T11:20:55.864 INFO:teuthology.orchestra.run.smithi136.stdout:55834574859 2024-04-25T11:20:55.865 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:55.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T11:20:55.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T11:20:55.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 --> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fae50156960 con 0x7fae30078c10 2024-04-25T11:20:55.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae4cff9640 1 -- 172.21.15.136:0/682996041 <== osd.6 v2:172.21.15.157:6824/224466234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fae50156960 con 0x7fae30078c10 2024-04-25T11:20:55.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7fae30078c10 msgr2=0x7fae3007b0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7fae30078c10 0x7fae3007b0a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae3006f450 msgr2=0x7fae30071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae3006f450 0x7fae30071960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae400098f0 tx=0x7fae40009190 comp rx=0 tx=0).stop 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 msgr2=0x7fae5015acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 --2- 172.21.15.136:0/682996041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae5015ca40 0x7fae5015acb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fae4801ade0 tx=0x7fae480194d0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae4ffff640 1 -- 172.21.15.136:0/682996041 reap_dead start 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 shutdown_connections 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 >> 172.21.15.136:0/682996041 conn(0x7fae5006d700 msgr2=0x7fae5015d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 shutdown_connections 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fae45ffb640 1 -- 172.21.15.136:0/682996041 wait complete. 2024-04-25T11:20:55.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 --> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd4b404fcb0 con 0x7fd4a0078b70 2024-04-25T11:20:55.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.848+0000 7fd4b8ff9640 1 -- 172.21.15.136:0/652678266 <== osd.7 v2:172.21.15.157:6809/498117293 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd4b404fcb0 con 0x7fd4a0078b70 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7fd4a0078b70 msgr2=0x7fd4a007b000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7fd4a0078b70 0x7fd4a007b000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd4a006f430 msgr2=0x7fd4a0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd4a006f430 0x7fd4a0071940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000abb0 tx=0x7fd4b0008040 comp rx=0 tx=0).stop 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 msgr2=0x7fd4b40cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 --2- 172.21.15.136:0/652678266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4b4036c50 0x7fd4b40cfdc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0520f0 tx=0x7fd4bc06aca0 comp rx=0 tx=0).stop 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c0d38640 1 -- 172.21.15.136:0/652678266 reap_dead start 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 shutdown_connections 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 >> 172.21.15.136:0/652678266 conn(0x7fd4b40514d0 msgr2=0x7fd4b404c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 shutdown_connections 2024-04-25T11:20:55.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:55.852+0000 7fd4c1539640 1 -- 172.21.15.136:0/652678266 wait complete. 2024-04-25T11:20:55.871 INFO:teuthology.orchestra.run.smithi136.stdout:55834574862 2024-04-25T11:20:55.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T11:20:55.872 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:55.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T11:20:55.873 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:55.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T11:20:55.874 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:55.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T11:20:55.876 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:55.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T11:20:55.877 INFO:teuthology.orchestra.run.smithi136.stdout:55834574862 2024-04-25T11:20:55.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T11:20:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.068+0000 7f6fd463f640 1 Processor -- start 2024-04-25T11:20:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.068+0000 7f6fd463f640 1 -- start start 2024-04-25T11:20:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd463f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6fcc073970 0x7f6fcc166260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd463f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd463f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fcc16d060 con 0x7f6fcc165e50 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd463f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6fcc1667a0 con 0x7f6fcc073970 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd463f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc08d2b0 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45178/0 (socket says 172.21.15.136:45178) 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 -- 172.21.15.136:0/742985708 learned_addr learned my addr 172.21.15.136:0/742985708 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd1bda640 1 --2- 172.21.15.136:0/742985708 >> v2:172.21.15.157:3300/0 conn(0x7f6fcc073970 0x7f6fcc166260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f03f995b640 1 Processor -- start 2024-04-25T11:20:56.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f03f995b640 1 -- start start 2024-04-25T11:20:56.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f995b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f40c7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f995b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 0x7f03f41646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f995b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03f40730c0 con 0x7f03f40c92c0 2024-04-25T11:20:56.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f995b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f03f40733a0 con 0x7f03f40c7e60 2024-04-25T11:20:56.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f995b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03f4073680 con 0x7f03f40c8eb0 2024-04-25T11:20:56.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f03f40c92c0 0x7f03f415d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49672/0 (socket says 172.21.15.136:49672) 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f3fff640 1 -- 172.21.15.136:0/462367744 learned_addr learned my addr 172.21.15.136:0/462367744 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f915a640 1 --2- 172.21.15.136:0/462367744 >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 0x7f03f41646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f8959640 1 --2- 172.21.15.136:0/462367744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f40c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd13d9640 1 -- 172.21.15.136:0/742985708 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873163528 0 0) 0x7f6fcc16d060 con 0x7f6fcc165e50 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 -- 172.21.15.136:0/742985708 >> v2:172.21.15.157:3300/0 conn(0x7f6fcc073970 msgr2=0x7f6fcc166260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 --2- 172.21.15.136:0/742985708 >> v2:172.21.15.157:3300/0 conn(0x7f6fcc073970 0x7f6fcc166260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 -- 172.21.15.136:0/742985708 >> v1:172.21.15.157:6789/0 conn(0x7f6fcc165e50 legacy=0x7f6fcc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.072+0000 7f6fd2bdc640 1 -- 172.21.15.136:0/742985708 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc08d840 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f37fe640 1 -- 172.21.15.136:0/462367744 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308847795 0 0) 0x7f03f40730c0 con 0x7f03f40c92c0 2024-04-25T11:20:56.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f37fe640 1 -- 172.21.15.136:0/462367744 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03e4003530 con 0x7f03f40c92c0 2024-04-25T11:20:56.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f8959640 1 -- 172.21.15.136:0/462367744 >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 msgr2=0x7f03f41646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f8959640 1 --2- 172.21.15.136:0/462367744 >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 0x7f03f41646c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd2bdc640 1 --2- 172.21.15.136:0/742985708 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400d370 tx=0x7f6fc401edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7641b8b5d0864828 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f8959640 1 -- 172.21.15.136:0/462367744 >> v1:172.21.15.136:6789/0 conn(0x7f03f40c92c0 legacy=0x7f03f415d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f8959640 1 -- 172.21.15.136:0/462367744 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f4073960 con 0x7f03f40c8eb0 2024-04-25T11:20:56.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f03f915a640 1 --2- 172.21.15.136:0/462367744 >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 0x7f03f41646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:56.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd13d9640 1 -- 172.21.15.136:0/742985708 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc40ad070 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd13d9640 1 -- 172.21.15.136:0/742985708 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc40039a0 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd13d9640 1 -- 172.21.15.136:0/742985708 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400be60 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7fb2a5235640 1 Processor -- start 2024-04-25T11:20:56.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7fb2a5235640 1 -- start start 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 -- 172.21.15.136:0/742985708 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 msgr2=0x7f6fcc070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 --2- 172.21.15.136:0/742985708 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400d370 tx=0x7f6fc401edf0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 -- 172.21.15.136:0/742985708 shutdown_connections 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 --2- 172.21.15.136:0/742985708 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc06fcf0 0x7f6fcc070100 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 --2- 172.21.15.136:0/742985708 >> v2:172.21.15.157:3300/0 conn(0x7f6fcc073970 0x7f6fcc166260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 -- 172.21.15.136:0/742985708 >> 172.21.15.136:0/742985708 conn(0x7f6fcc0c9d50 msgr2=0x7f6fcc0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 -- 172.21.15.136:0/742985708 shutdown_connections 2024-04-25T11:20:56.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.076+0000 7f6fd463f640 1 -- 172.21.15.136:0/742985708 wait complete. 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb2a5235640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 0x7fb2a0160000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb2a5235640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a006dcc0 con 0x7fb2a006ea60 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb2a5235640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a00caa30 con 0x7fb2a00cbe80 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb2a5235640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a006daa0 con 0x7fb2a006df90 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb2a006ea60 0x7fb2a015c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49676/0 (socket says 172.21.15.136:49676) 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29e59c640 1 -- 172.21.15.136:0/972398213 learned_addr learned my addr 172.21.15.136:0/972398213 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:56.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29ed9d640 1 --2- 172.21.15.136:0/972398213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 0x7fb2a0160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7f03f8959640 1 --2- 172.21.15.136:0/462367744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f40c7920 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f03e80097c0 tx=0x7f03e801c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2425345e7c8e2731 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098241393 0 0) 0x7fb2a006dcc0 con 0x7fb2a006ea60 2024-04-25T11:20:56.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb288003530 con 0x7fb2a006ea60 2024-04-25T11:20:56.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090329352 0 0) 0x7fb2a00caa30 con 0x7fb2a00cbe80 2024-04-25T11:20:56.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2a006dcc0 con 0x7fb2a00cbe80 2024-04-25T11:20:56.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29ed9d640 1 -- 172.21.15.136:0/972398213 >> v1:172.21.15.157:6789/0 conn(0x7fb2a00cbe80 legacy=0x7fb2a0166920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29ed9d640 1 -- 172.21.15.136:0/972398213 >> v1:172.21.15.136:6789/0 conn(0x7fb2a006ea60 legacy=0x7fb2a015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7fb29ed9d640 1 -- 172.21.15.136:0/972398213 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a00ca470 con 0x7fb2a006df90 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7f6fd463f640 1 Processor -- start 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.080+0000 7f6fd463f640 1 -- start start 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 0x7f6fcc16a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 0x7f6fcc15f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd23db640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 0x7f6fcc16a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd23db640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 0x7f6fcc16a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41106/0 (socket says 172.21.15.136:41106) 2024-04-25T11:20:56.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb29ed9d640 1 --2- 172.21.15.136:0/972398213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 0x7fb2a0160000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097f0 tx=0x7fb29001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e211c2bc6fe2048 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45202/0 (socket says 172.21.15.136:45202) 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd2bdc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 0x7f6fcc15f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd2bdc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 0x7f6fcc15f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35060/0 (socket says 172.21.15.136:35060) 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd23db640 1 -- 172.21.15.136:0/3749534251 learned_addr learned my addr 172.21.15.136:0/3749534251 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2900b1070 con 0x7fb2a006df90 2024-04-25T11:20:56.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc175f40 con 0x7f6fcc165e50 2024-04-25T11:20:56.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc175dc0 con 0x7f6fcc06fcf0 2024-04-25T11:20:56.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f37fe640 1 -- 172.21.15.136:0/462367744 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e80b0070 con 0x7f03f40c8eb0 2024-04-25T11:20:56.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc1760c0 con 0x7f6fcc073970 2024-04-25T11:20:56.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f37fe640 1 -- 172.21.15.136:0/462367744 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e80ab400 con 0x7f03f40c8eb0 2024-04-25T11:20:56.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2900ac630 con 0x7fb2a006df90 2024-04-25T11:20:56.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb29dd9b640 1 -- 172.21.15.136:0/972398213 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2900a8050 con 0x7fb2a006df90 2024-04-25T11:20:56.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 -- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 msgr2=0x7f6fcc16a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 0x7f6fcc16a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 -- 172.21.15.136:0/3749534251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 msgr2=0x7f6fcc15f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 0x7f6fcc15f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc16a880 con 0x7f6fcc073970 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f37fe640 1 -- 172.21.15.136:0/462367744 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0001680 con 0x7f03f40c8eb0 2024-04-25T11:20:56.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fd1bda640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e870 tx=0x7f6fbc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 -- 172.21.15.136:0/462367744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 msgr2=0x7f03f40c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 --2- 172.21.15.136:0/462367744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f40c7920 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f03e80097c0 tx=0x7f03e801c2e0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 -- 172.21.15.136:0/972398213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 msgr2=0x7fb2a0160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 --2- 172.21.15.136:0/972398213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 0x7fb2a0160000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097f0 tx=0x7fb29001c4e0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 -- 172.21.15.136:0/462367744 shutdown_connections 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 --2- 172.21.15.136:0/462367744 >> v2:172.21.15.157:3300/0 conn(0x7f03f40c7e60 0x7f03f41646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 --2- 172.21.15.136:0/462367744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f40c7920 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f03f995b640 1 -- 172.21.15.136:0/462367744 >> 172.21.15.136:0/462367744 conn(0x7f03f406d100 msgr2=0x7f03f4070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 -- 172.21.15.136:0/972398213 shutdown_connections 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 --2- 172.21.15.136:0/972398213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006df90 0x7fb2a0160000 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 -- 172.21.15.136:0/972398213 >> 172.21.15.136:0/972398213 conn(0x7fb2a0072790 msgr2=0x7fb2a0072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc017070 con 0x7f6fcc073970 2024-04-25T11:20:56.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.084+0000 7fb2a5235640 1 -- 172.21.15.136:0/972398213 shutdown_connections 2024-04-25T11:20:56.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7fb2a5235640 1 -- 172.21.15.136:0/972398213 wait complete. 2024-04-25T11:20:56.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f03f995b640 1 -- 172.21.15.136:0/462367744 shutdown_connections 2024-04-25T11:20:56.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc20bb50 con 0x7f6fcc073970 2024-04-25T11:20:56.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc169860 con 0x7f6fcc073970 2024-04-25T11:20:56.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00ee60 con 0x7f6fcc073970 2024-04-25T11:20:56.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00cb60 con 0x7f6fcc073970 2024-04-25T11:20:56.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7fb2a5235640 1 Processor -- start 2024-04-25T11:20:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f03f995b640 1 -- 172.21.15.136:0/462367744 wait complete. 2024-04-25T11:20:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f03f995b640 1 Processor -- start 2024-04-25T11:20:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7fb2a5235640 1 -- start start 2024-04-25T11:20:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6fbc01a070 con 0x7f6fcc073970 2024-04-25T11:20:56.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f6fc2ffd640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 0x7f6fb0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.088+0000 7f03f995b640 1 -- start start 2024-04-25T11:20:56.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6fbc08f7c0 con 0x7f6fcc073970 2024-04-25T11:20:56.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f6fd23db640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 0x7f6fb0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2a006df90 0x7fb2a00cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006ea60 0x7fb2a0154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fb40052a0 con 0x7f6fcc073970 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03f40c7e60 0x7f03f40c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f4205a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4070da0 con 0x7f03f40c92c0 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03f4073c40 con 0x7f03f40c7e60 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03f4070f20 con 0x7f03f40c8eb0 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a006ee70 con 0x7fb2a00cbe80 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a00cc6a0 con 0x7fb2a006df90 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a006eff0 con 0x7fb2a006ea60 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35066/0 (socket says 172.21.15.136:35066) 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 -- 172.21.15.136:0/594568908 learned_addr learned my addr 172.21.15.136:0/594568908 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29ed9d640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2a006df90 0x7fb2a00cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f6fd23db640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 0x7f6fb0071910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8001840 tx=0x7f6fc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35068/0 (socket says 172.21.15.136:35068) 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 -- 172.21.15.136:0/2909499870 learned_addr learned my addr 172.21.15.136:0/2909499870 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f8959640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03f40c7e60 0x7f03f40c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f3fff640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f4205a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29e59c640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006ea60 0x7fb2a0154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 -- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006ea60 msgr2=0x7fb2a0154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006ea60 0x7fb2a0154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 -- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2a006df90 msgr2=0x7fb2a00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2a006df90 0x7fb2a00cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 -- 172.21.15.136:0/594568908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a0157df0 con 0x7fb2a00cbe80 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 -- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 msgr2=0x7f03f4205a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f4205a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 -- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03f40c7e60 msgr2=0x7f03f40c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03f40c7e60 0x7f03f40c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 -- 172.21.15.136:0/2909499870 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f40c7100 con 0x7f03f40c92c0 2024-04-25T11:20:56.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb29f59e640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c3f0 tx=0x7fb294001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f915a640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009e60 tx=0x7f03ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec018030 con 0x7f03f40c92c0 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ec00cdf0 con 0x7f03f40c92c0 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec003bc0 con 0x7f03f40c92c0 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294017070 con 0x7fb2a00cbe80 2024-04-25T11:20:56.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2940052f0 con 0x7fb2a00cbe80 2024-04-25T11:20:56.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a0157b50 con 0x7fb2a00cbe80 2024-04-25T11:20:56.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294003840 con 0x7fb2a00cbe80 2024-04-25T11:20:56.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 -- 172.21.15.136:0/2909499870 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f40cbe80 con 0x7f03f40c92c0 2024-04-25T11:20:56.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7f03f995b640 1 -- 172.21.15.136:0/2909499870 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f40c63c0 con 0x7f03f40c92c0 2024-04-25T11:20:56.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.092+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a0158dc0 con 0x7fb2a00cbe80 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7f03f995b640 1 -- 172.21.15.136:0/2909499870 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f406a2e0 con 0x7f03f40c92c0 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f03ec01b070 con 0x7f03f40c92c0 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7f03f17fa640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 0x7f03d80719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f03ec08fdb0 con 0x7f03f40c92c0 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7f03f8959640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 0x7f03d80719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a0159200 con 0x7fb2a00cbe80 2024-04-25T11:20:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.096+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb29401a070 con 0x7fb2a00cbe80 2024-04-25T11:20:56.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fb2877fe640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 0x7fb28c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7f03f8959640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 0x7f03d80719e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f03e80ad000 tx=0x7f03e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb29408e850 con 0x7fb2a00cbe80 2024-04-25T11:20:56.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fb29ed9d640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 0x7fb28c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 Processor -- start 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 -- start start 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7fc0cc4b0 0x7ff7fc0cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff801feb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff7fc156010 0x7ff7fc06ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:53136/0 (socket says 172.21.15.136:53136) 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff801feb640 1 -- 172.21.15.136:0/3373086019 learned_addr learned my addr 172.21.15.136:0/3373086019 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8027ec640 1 --2- 172.21.15.136:0/3373086019 >> v2:172.21.15.136:3300/0 conn(0x7ff7fc0cc4b0 0x7ff7fc0cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc15ca90 con 0x7ff7fc156010 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8017ea640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff7fc0b4300 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc0b4020 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff800fe9640 1 -- 172.21.15.136:0/3373086019 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 670494435 0 0) 0x7ff7fc15ca90 con 0x7ff7fc156010 2024-04-25T11:20:56.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fa8c603f640 1 Processor -- start 2024-04-25T11:20:56.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff800fe9640 1 -- 172.21.15.136:0/3373086019 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7e8003530 con 0x7ff7fc156010 2024-04-25T11:20:56.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fb29ed9d640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 0x7fb28c071940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb2900ae000 tx=0x7fb2900a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7fa8c603f640 1 -- start start 2024-04-25T11:20:56.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6fbc064350 con 0x7f6fcc073970 2024-04-25T11:20:56.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8c603f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 0x7fa8b804c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8c603f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 0x7fa8b80e60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8c603f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b80d5e20 con 0x7fa8b8046b00 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8c603f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b80460a0 con 0x7fa8b8047a60 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8c603f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b804d230 con 0x7fa8b804c930 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8beffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 0x7fa8b804c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8beffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 0x7fa8b804c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41142/0 (socket says 172.21.15.136:41142) 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8beffd640 1 -- 172.21.15.136:0/1291785123 learned_addr learned my addr 172.21.15.136:0/1291785123 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8027ec640 1 -- 172.21.15.136:0/3373086019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 msgr2=0x7ff7fc0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8027ec640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc0cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8027ec640 1 -- 172.21.15.136:0/3373086019 >> v1:172.21.15.157:6789/0 conn(0x7ff7fc156010 legacy=0x7ff7fc06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.100+0000 7ff8027ec640 1 -- 172.21.15.136:0/3373086019 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc15cd30 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8bffff640 1 --2- 172.21.15.136:0/1291785123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 0x7fa8b80e60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7ff8027ec640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc0cc8c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009d90 tx=0x7ff7f401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e41b5bfbf01e551 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8bffff640 1 -- 172.21.15.136:0/1291785123 >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 msgr2=0x7fa8b804c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8bffff640 1 --2- 172.21.15.136:0/1291785123 >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 0x7fa8b804c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8bffff640 1 -- 172.21.15.136:0/1291785123 >> v1:172.21.15.136:6789/0 conn(0x7fa8b8046b00 legacy=0x7fa8b80dc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.104+0000 7fa8bffff640 1 -- 172.21.15.136:0/1291785123 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b80466c0 con 0x7fa8b804c930 2024-04-25T11:20:56.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.108+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f03ec022070 con 0x7f03f40c92c0 2024-04-25T11:20:56.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.108+0000 7fa8bffff640 1 --2- 172.21.15.136:0/1291785123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 0x7fa8b80e60b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8046220 tx=0x7fa8c009c020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e6fd9b1c8d0091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.108+0000 7fa8be7fc640 1 -- 172.21.15.136:0/1291785123 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0125d00 con 0x7fa8b804c930 2024-04-25T11:20:56.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.108+0000 7fa8be7fc640 1 -- 172.21.15.136:0/1291785123 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c012d440 con 0x7fa8b804c930 2024-04-25T11:20:56.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.108+0000 7fa8be7fc640 1 -- 172.21.15.136:0/1291785123 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c012d8b0 con 0x7fa8b804c930 2024-04-25T11:20:56.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 -- 172.21.15.136:0/1291785123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 msgr2=0x7fa8b80e60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 --2- 172.21.15.136:0/1291785123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 0x7fa8b80e60b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8046220 tx=0x7fa8c009c020 comp rx=0 tx=0).stop 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 -- 172.21.15.136:0/1291785123 shutdown_connections 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 --2- 172.21.15.136:0/1291785123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b804c930 0x7fa8b80e60b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 --2- 172.21.15.136:0/1291785123 >> v2:172.21.15.157:3300/0 conn(0x7fa8b8047a60 0x7fa8b804c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 -- 172.21.15.136:0/1291785123 >> 172.21.15.136:0/1291785123 conn(0x7fa8b8050c80 msgr2=0x7fa8b8051090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 -- 172.21.15.136:0/1291785123 shutdown_connections 2024-04-25T11:20:56.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.112+0000 7fa8c603f640 1 -- 172.21.15.136:0/1291785123 wait complete. 2024-04-25T11:20:56.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7fa8c603f640 1 Processor -- start 2024-04-25T11:20:56.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7fa8c603f640 1 -- start start 2024-04-25T11:20:56.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7ff800fe9640 1 -- 172.21.15.136:0/3373086019 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40b4070 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7ff800fe9640 1 -- 172.21.15.136:0/3373086019 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f40af600 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb29405ae30 con 0x7fb2a00cbe80 2024-04-25T11:20:56.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.116+0000 7ff800fe9640 1 -- 172.21.15.136:0/3373086019 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40af9b0 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.120+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cc4b0 msgr2=0x7ff7fc0cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.120+0000 7ff80424f640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc0cc8c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009d90 tx=0x7ff7f401f150 comp rx=0 tx=0).stop 2024-04-25T11:20:56.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 shutdown_connections 2024-04-25T11:20:56.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc0cc8c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 --2- 172.21.15.136:0/3373086019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 >> 172.21.15.136:0/3373086019 conn(0x7ff7fc074700 msgr2=0x7ff7fc0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 shutdown_connections 2024-04-25T11:20:56.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 -- 172.21.15.136:0/3373086019 wait complete. 2024-04-25T11:20:56.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.124+0000 7ff80424f640 1 Processor -- start 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b8046b00 0x7fa8b81718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b804c930 0x7fa8b80cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b80dbc20 con 0x7fa8b8047a60 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b80db940 con 0x7fa8b804c930 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8c603f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b8033830 con 0x7fa8b8046b00 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35092/0 (socket says 172.21.15.136:35092) 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 -- 172.21.15.136:0/3205529773 learned_addr learned my addr 172.21.15.136:0/3205529773 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8bffff640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b804c930 0x7fa8b80cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8bf7fe640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b8046b00 0x7fa8b81718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 -- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b8046b00 msgr2=0x7fa8b81718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b8046b00 0x7fa8b81718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 -- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b804c930 msgr2=0x7fa8b80cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b804c930 0x7fa8b80cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 -- 172.21.15.136:0/3205529773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b80d9a90 con 0x7fa8b8047a60 2024-04-25T11:20:56.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8beffd640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e710 tx=0x7fa8b400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4016070 con 0x7fa8b8047a60 2024-04-25T11:20:56.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b400c9d0 con 0x7fa8b8047a60 2024-04-25T11:20:56.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400b040 con 0x7fa8b8047a60 2024-04-25T11:20:56.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.128+0000 7ff80424f640 1 -- start start 2024-04-25T11:20:56.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b80cda60 con 0x7fa8b8047a60 2024-04-25T11:20:56.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b81752c0 con 0x7fa8b8047a60 2024-04-25T11:20:56.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fd87b378640 1 Processor -- start 2024-04-25T11:20:56.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fd87b378640 1 -- start start 2024-04-25T11:20:56.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8b80d9d50 con 0x7fa8b8047a60 2024-04-25T11:20:56.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa8b4019070 con 0x7fa8b8047a60 2024-04-25T11:20:56.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8abfff640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa8b408eeb0 con 0x7fa8b8047a60 2024-04-25T11:20:56.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7fa8bf7fe640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fa8bf7fe640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0098e0 tx=0x7fa8ac009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87b378640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd86c0d6220 0x7fd86c0df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87b378640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd86c046890 con 0x7fd86c0092d0 2024-04-25T11:20:56.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87b378640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd86c0e0190 con 0x7fd86c0dfaa0 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87b378640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd86c0e0470 con 0x7fd86c0d6220 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd86c0d6220 0x7fd86c0df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd86c0d6220 0x7fd86c0df560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41180/0 (socket says 172.21.15.136:41180) 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 -- 172.21.15.136:0/1311740256 learned_addr learned my addr 172.21.15.136:0/1311740256 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 -- 172.21.15.136:0/1311740256 >> v1:172.21.15.157:6789/0 conn(0x7fd86c0dfaa0 legacy=0x7fd86c008bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 -- 172.21.15.136:0/1311740256 >> v1:172.21.15.136:6789/0 conn(0x7fd86c0092d0 legacy=0x7fd86c0e68f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:20:56.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7fd87a376640 1 -- 172.21.15.136:0/1311740256 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c04cd40 con 0x7fd86c0d6220 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc1530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc156010 0x7ff7fc1f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0cce00 con 0x7ff7fc156010 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0cbf90 con 0x7ff7fc0cc4b0 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.132+0000 7ff80424f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc06d570 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff8017ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc1530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45258/0 (socket says 172.21.15.136:45258) 2024-04-25T11:20:56.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 -- 172.21.15.136:0/1908725816 learned_addr learned my addr 172.21.15.136:0/1908725816 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 -- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 msgr2=0x7ff7fc1530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc1530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 -- 172.21.15.136:0/1908725816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc156010 msgr2=0x7ff7fc1f58e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc156010 0x7ff7fc1f58e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 -- 172.21.15.136:0/1908725816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc15d4c0 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff8017ea640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc1530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff801feb640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80010f0 tx=0x7ff7f801d8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8028070 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f8005af0 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc150860 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80070d0 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc14f290 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.136+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff7f802b070 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7ff7f27fc640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 0x7ff7e0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff7f800e020 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7ff8017ea640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 0x7ff7e0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87a376640 1 --2- 172.21.15.136:0/1311740256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0d6220 0x7fd86c0df560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd870009870 tx=0x7fd87001e5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11e1af33a7e45361 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd879374640 1 -- 172.21.15.136:0/1311740256 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700b2070 con 0x7fd86c0d6220 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd879374640 1 -- 172.21.15.136:0/1311740256 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8700ad600 con 0x7fd86c0d6220 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd879374640 1 -- 172.21.15.136:0/1311740256 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700ad9d0 con 0x7fd86c0d6220 2024-04-25T11:20:56.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7ff8017ea640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 0x7ff7e0071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0ccae0 tx=0x7ff7ec009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 -- 172.21.15.136:0/1311740256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0d6220 msgr2=0x7fd86c0df560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 --2- 172.21.15.136:0/1311740256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0d6220 0x7fd86c0df560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd870009870 tx=0x7fd87001e5d0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7fc06a2e0 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 -- 172.21.15.136:0/1311740256 shutdown_connections 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 --2- 172.21.15.136:0/1311740256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0d6220 0x7fd86c0df560 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 -- 172.21.15.136:0/1311740256 >> 172.21.15.136:0/1311740256 conn(0x7fd86c0514d0 msgr2=0x7fd86c0518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 -- 172.21.15.136:0/1311740256 shutdown_connections 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fd87b378640 1 -- 172.21.15.136:0/1311740256 wait complete. 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.140+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa8b405b4a0 con 0x7fa8b8047a60 2024-04-25T11:20:56.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 Processor -- start 2024-04-25T11:20:56.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 -- start start 2024-04-25T11:20:56.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd86c0d6220 0x7fd86c0de560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0dfaa0 0x7fd86c0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c0465f0 con 0x7fd86c0092d0 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87a376640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c046470 con 0x7fd86c0dfaa0 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87b378640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd86c034060 con 0x7fd86c0d6220 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87a376640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35098/0 (socket says 172.21.15.136:35098) 2024-04-25T11:20:56.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87a376640 1 -- 172.21.15.136:0/1082888822 learned_addr learned my addr 172.21.15.136:0/1082888822 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd87ab77640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0dfaa0 0x7fd86c0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.144+0000 7fd879b75640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd86c0d6220 0x7fd86c0de560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 -- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd86c0d6220 msgr2=0x7fd86c0de560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd86c0d6220 0x7fd86c0de560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 -- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0dfaa0 msgr2=0x7fd86c0cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0dfaa0 0x7fd86c0cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 -- 172.21.15.136:0/1082888822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c0dac40 con 0x7fd86c0092d0 2024-04-25T11:20:56.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87a376640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd87001c200 tx=0x7fd87001b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700b2040 con 0x7fd86c0092d0 2024-04-25T11:20:56.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8700bb070 con 0x7fd86c0092d0 2024-04-25T11:20:56.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87b378640 1 -- 172.21.15.136:0/1082888822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86c0d9680 con 0x7fd86c0092d0 2024-04-25T11:20:56.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700ac9a0 con 0x7fd86c0092d0 2024-04-25T11:20:56.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd87b378640 1 -- 172.21.15.136:0/1082888822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86c0d9f00 con 0x7fd86c0092d0 2024-04-25T11:20:56.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd8700acb40 con 0x7fd86c0092d0 2024-04-25T11:20:56.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8500052a0 con 0x7fd86c0092d0 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7f3034a10640 1 Processor -- start 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7f3034a10640 1 -- start start 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f3034a10640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f30280e4030 0x7f3028007c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f3034a10640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30280e4440 0x7f3028008190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f3034a10640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3028046cf0 con 0x7f3028006d50 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f3034a10640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3028046fd0 con 0x7f30280e4030 2024-04-25T11:20:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f3034a10640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f30280472b0 con 0x7f30280e4440 2024-04-25T11:20:56.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 0x7fd858071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd87012c020 con 0x7fd86c0092d0 2024-04-25T11:20:56.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.148+0000 7fd879b75640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 0x7fd858071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7fd879b75640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 0x7fd858071940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd86c005920 tx=0x7fd864009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302f59e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3028006d50 0x7f30280e6d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49708/0 (socket says 172.21.15.136:49708) 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302f59e640 1 -- 172.21.15.136:0/2025660741 learned_addr learned my addr 172.21.15.136:0/2025660741 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302ed9d640 1 --2- 172.21.15.136:0/2025660741 >> v2:172.21.15.157:3300/0 conn(0x7f30280e4030 0x7f3028007c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302ed9d640 1 -- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30280e4440 msgr2=0x7f3028008190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302ed9d640 1 --2- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30280e4440 0x7f3028008190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302ed9d640 1 -- 172.21.15.136:0/2025660741 >> v1:172.21.15.136:6789/0 conn(0x7f3028006d50 legacy=0x7f30280e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7f302ed9d640 1 -- 172.21.15.136:0/2025660741 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30280e3830 con 0x7f30280e4030 2024-04-25T11:20:56.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.152+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff7f805b020 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f302ed9d640 1 --2- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4030 0x7f3028007c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30300515a0 tx=0x7f30300997a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834ef2dedb1c620c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f302dd9b640 1 -- 172.21.15.136:0/2025660741 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30301256b0 con 0x7f30280e4030 2024-04-25T11:20:56.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f302dd9b640 1 -- 172.21.15.136:0/2025660741 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3030125890 con 0x7f30280e4030 2024-04-25T11:20:56.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f302dd9b640 1 -- 172.21.15.136:0/2025660741 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030125d00 con 0x7f30280e4030 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 -- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4030 msgr2=0x7f3028007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 --2- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4030 0x7f3028007c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f30300515a0 tx=0x7f30300997a0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 -- 172.21.15.136:0/2025660741 shutdown_connections 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 --2- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f30280e4440 0x7f3028008190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 --2- 172.21.15.136:0/2025660741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4030 0x7f3028007c50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 -- 172.21.15.136:0/2025660741 >> 172.21.15.136:0/2025660741 conn(0x7f3028050c50 msgr2=0x7f3028051060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 -- 172.21.15.136:0/2025660741 shutdown_connections 2024-04-25T11:20:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.156+0000 7f3034a10640 1 -- 172.21.15.136:0/2025660741 wait complete. 2024-04-25T11:20:56.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd8700182e0 con 0x7fd86c0092d0 2024-04-25T11:20:56.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7f3034a10640 1 Processor -- start 2024-04-25T11:20:56.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7f79672b3640 1 Processor -- start 2024-04-25T11:20:56.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7f79672b3640 1 -- start start 2024-04-25T11:20:56.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7f3034a10640 1 -- start start 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.160+0000 7f3034a10640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3028006d50 0x7f3028181740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4440 0x7f30280488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3028046910 con 0x7f302804a1d0 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3028007570 con 0x7f30280e4440 2024-04-25T11:20:56.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3028046a90 con 0x7f3028006d50 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35110/0 (socket says 172.21.15.136:35110) 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 -- 172.21.15.136:0/3375624186 learned_addr learned my addr 172.21.15.136:0/3375624186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302ed9d640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3028006d50 0x7f3028181740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 -- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3028006d50 msgr2=0x7f3028181740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3028006d50 0x7f3028181740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 -- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4440 msgr2=0x7f30280488c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4440 0x7f30280488c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 -- 172.21.15.136:0/3375624186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302817d690 con 0x7f302804a1d0 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f302f59e640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f302000c3f0 tx=0x7f302000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020017070 con 0x7f302804a1d0 2024-04-25T11:20:56.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30280d7840 con 0x7f302804a1d0 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f79672b3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f79580060a0 0x7f79580e07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f79672b3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795804b5e0 con 0x7f79580064b0 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f79672b3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795804b850 con 0x7f79580360c0 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f79672b3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f795804bb30 con 0x7f79580060a0 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f79580060a0 0x7f79580e07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f79580060a0 0x7f79580e07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41222/0 (socket says 172.21.15.136:41222) 2024-04-25T11:20:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 -- 172.21.15.136:0/723474592 learned_addr learned my addr 172.21.15.136:0/723474592 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 -- 172.21.15.136:0/723474592 >> v1:172.21.15.157:6789/0 conn(0x7f79580360c0 legacy=0x7f79580d7350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:20:56.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 -- 172.21.15.136:0/723474592 >> v1:172.21.15.136:6789/0 conn(0x7f79580064b0 legacy=0x7f79580e0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.164+0000 7f796484e640 1 -- 172.21.15.136:0/723474592 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7958047380 con 0x7f79580060a0 2024-04-25T11:20:56.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3028049230 con 0x7f302804a1d0 2024-04-25T11:20:56.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3020005cf0 con 0x7f302804a1d0 2024-04-25T11:20:56.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30200072f0 con 0x7f302804a1d0 2024-04-25T11:20:56.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f302001a070 con 0x7f302804a1d0 2024-04-25T11:20:56.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f796484e640 1 --2- 172.21.15.136:0/723474592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f79580e07d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f795c009870 tx=0x7f795c01c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06055b687b8907c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f7957fff640 1 -- 172.21.15.136:0/723474592 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0aa650 con 0x7f79580060a0 2024-04-25T11:20:56.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f30177fe640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 0x7f301c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3020093060 con 0x7f302804a1d0 2024-04-25T11:20:56.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f302ed9d640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 0x7f301c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f7957fff640 1 -- 172.21.15.136:0/723474592 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795c0af070 con 0x7f79580060a0 2024-04-25T11:20:56.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f7957fff640 1 -- 172.21.15.136:0/723474592 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0aaa10 con 0x7f79580060a0 2024-04-25T11:20:56.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.168+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3028048fa0 con 0x7f302804a1d0 2024-04-25T11:20:56.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.172+0000 7f79672b3640 1 -- 172.21.15.136:0/723474592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 msgr2=0x7f79580e07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.172+0000 7f79672b3640 1 --2- 172.21.15.136:0/723474592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f79580e07d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f795c009870 tx=0x7f795c01c4f0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f302005b390 con 0x7f302804a1d0 2024-04-25T11:20:56.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 -- 172.21.15.136:0/723474592 shutdown_connections 2024-04-25T11:20:56.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 --2- 172.21.15.136:0/723474592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f79580e07d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 -- 172.21.15.136:0/723474592 >> 172.21.15.136:0/723474592 conn(0x7f79580503c0 msgr2=0x7f79580507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 -- 172.21.15.136:0/723474592 shutdown_connections 2024-04-25T11:20:56.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f302ed9d640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 0x7f301c071a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f302804d160 tx=0x7f303006f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 -- 172.21.15.136:0/723474592 wait complete. 2024-04-25T11:20:56.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 Processor -- start 2024-04-25T11:20:56.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.176+0000 7f79672b3640 1 -- start start 2024-04-25T11:20:56.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f795804f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 0x7f79580c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 0x7f79580df1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796504f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f795804f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796504f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f795804f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41238/0 (socket says 172.21.15.136:41238) 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f7965850640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 0x7f79580df1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f7965850640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 0x7f79580df1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35116/0 (socket says 172.21.15.136:35116) 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 0x7f79580c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796504f640 1 -- 172.21.15.136:0/3552900620 learned_addr learned my addr 172.21.15.136:0/3552900620 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f79580350d0 con 0x7f79580360c0 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f795804bd90 con 0x7f79580060a0 2024-04-25T11:20:56.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f795804ffb0 con 0x7f79580064b0 2024-04-25T11:20:56.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 -- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 msgr2=0x7f795804f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f795804f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 -- 172.21.15.136:0/3552900620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 msgr2=0x7f79580df1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 0x7f79580df1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795804fae0 con 0x7f79580064b0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f796484e640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 0x7f79580c7940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7958051360 tx=0x7f795c01ad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0b3800 con 0x7f79580064b0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795c0a6040 con 0x7f79580064b0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.180+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79580d6350 con 0x7f79580064b0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c0b3c00 con 0x7f79580064b0 2024-04-25T11:20:56.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79580de5f0 con 0x7f79580064b0 2024-04-25T11:20:56.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f795c0c73a0 con 0x7f79580064b0 2024-04-25T11:20:56.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f7955ffb640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 0x7f7944071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:56.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f795c129d80 con 0x7f79580064b0 2024-04-25T11:20:56.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f796504f640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 0x7f7944071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:56.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.184+0000 7f796504f640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 0x7f7944071910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7950009a00 tx=0x7f7950009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:56.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.188+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7958005580 con 0x7f79580064b0 2024-04-25T11:20:56.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.200+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f795c016110 con 0x7f79580064b0 2024-04-25T11:20:56.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.432+0000 7f6fd463f640 1 -- 172.21.15.136:0/3749534251 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6fb4005590 con 0x7f6fcc073970 2024-04-25T11:20:56.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.432+0000 7f6fc2ffd640 1 -- 172.21.15.136:0/3749534251 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6fbc05bc80 con 0x7f6fcc073970 2024-04-25T11:20:56.432 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:56.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 msgr2=0x7f6fb0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 0x7f6fb0071910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8001840 tx=0x7f6fc8009040 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 msgr2=0x7f6fcc20b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00e870 tx=0x7f6fbc00ed30 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 shutdown_connections 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc165e50 0x7f6fcc15f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fb006f400 0x7f6fb0071910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fcc073970 0x7f6fcc20b610 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 --2- 172.21.15.136:0/3749534251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fcc06fcf0 0x7f6fcc16a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 >> 172.21.15.136:0/3749534251 conn(0x7f6fcc0c9d50 msgr2=0x7f6fcc0c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 shutdown_connections 2024-04-25T11:20:56.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.436+0000 7f6fc0ff9640 1 -- 172.21.15.136:0/3749534251 wait complete. 2024-04-25T11:20:56.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.448+0000 7f03f995b640 1 -- 172.21.15.136:0/2909499870 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f03f40c88b0 con 0x7f03f40c92c0 2024-04-25T11:20:56.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.448+0000 7f03f17fa640 1 -- 172.21.15.136:0/2909499870 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f03ec010040 con 0x7f03f40c92c0 2024-04-25T11:20:56.448 INFO:teuthology.orchestra.run.smithi136.stdout:55834574859 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 msgr2=0x7f03d80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 0x7f03d80719e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f03e80ad000 tx=0x7f03e8008040 comp rx=0 tx=0).stop 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 msgr2=0x7f03f4205f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009e60 tx=0x7f03ec00c490 comp rx=0 tx=0).stop 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 shutdown_connections 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03f40c92c0 0x7f03f4205f60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f03d806f4d0 0x7f03d80719e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03f40c8eb0 0x7f03f4205a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 --2- 172.21.15.136:0/2909499870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03f40c7e60 0x7f03f40c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 >> 172.21.15.136:0/2909499870 conn(0x7f03f406d100 msgr2=0x7f03f4151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 shutdown_connections 2024-04-25T11:20:56.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.460+0000 7f03deffd640 1 -- 172.21.15.136:0/2909499870 wait complete. 2024-04-25T11:20:56.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574861 for osd.4 2024-04-25T11:20:56.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.468+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff7fc0cd450 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.468+0000 7ff7f27fc640 1 -- 172.21.15.136:0/1908725816 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff7f805f320 con 0x7ff7fc0c61d0 2024-04-25T11:20:56.467 INFO:teuthology.orchestra.run.smithi136.stdout:55834574858 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 msgr2=0x7ff7e0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 0x7ff7e0071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0ccae0 tx=0x7ff7ec009190 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 msgr2=0x7ff7fc15d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80010f0 tx=0x7ff7f801d8a0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 shutdown_connections 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc156010 0x7ff7fc1f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc0cc4b0 0x7ff7fc1530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7e006f400 0x7ff7e0071910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 --2- 172.21.15.136:0/1908725816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc0c61d0 0x7ff7fc15d900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 >> 172.21.15.136:0/1908725816 conn(0x7ff7fc074700 msgr2=0x7ff7fc153e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 shutdown_connections 2024-04-25T11:20:56.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.472+0000 7ff80424f640 1 -- 172.21.15.136:0/1908725816 wait complete. 2024-04-25T11:20:56.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.476+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb2a006a2e0 con 0x7fb2a00cbe80 2024-04-25T11:20:56.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.476+0000 7fb2877fe640 1 -- 172.21.15.136:0/594568908 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb29405f130 con 0x7fb2a00cbe80 2024-04-25T11:20:56.474 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:56.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 msgr2=0x7fb28c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 0x7fb28c071940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb2900ae000 tx=0x7fb2900a7000 comp rx=0 tx=0).stop 2024-04-25T11:20:56.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 msgr2=0x7fb2a00c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c3f0 tx=0x7fb294001e80 comp rx=0 tx=0).stop 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 shutdown_connections 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2a00cbe80 0x7fb2a00c8d50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb28c06f430 0x7fb28c071940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2a006ea60 0x7fb2a0154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 --2- 172.21.15.136:0/594568908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2a006df90 0x7fb2a00cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 >> 172.21.15.136:0/594568908 conn(0x7fb2a0072790 msgr2=0x7fb2a00c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 shutdown_connections 2024-04-25T11:20:56.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.480+0000 7fb2a5235640 1 -- 172.21.15.136:0/594568908 wait complete. 2024-04-25T11:20:56.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.484+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa8b8005a00 con 0x7fa8b8047a60 2024-04-25T11:20:56.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.484+0000 7fa8abfff640 1 -- 172.21.15.136:0/3205529773 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa8b405f7a0 con 0x7fa8b8047a60 2024-04-25T11:20:56.483 INFO:teuthology.orchestra.run.smithi136.stdout:55834574859 2024-04-25T11:20:56.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 msgr2=0x7fa8a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0098e0 tx=0x7fa8ac009060 comp rx=0 tx=0).stop 2024-04-25T11:20:56.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 msgr2=0x7fa8b80d9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e710 tx=0x7fa8b400ebd0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 shutdown_connections 2024-04-25T11:20:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b804c930 0x7fa8b80cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8047a60 0x7fa8b80d9550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 --2- 172.21.15.136:0/3205529773 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8b8046b00 0x7fa8b81718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 >> 172.21.15.136:0/3205529773 conn(0x7fa8b8050c80 msgr2=0x7fa8b80dc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 shutdown_connections 2024-04-25T11:20:56.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fa8c603f640 1 -- 172.21.15.136:0/3205529773 wait complete. 2024-04-25T11:20:56.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.488+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd850005860 con 0x7fd86c0092d0 2024-04-25T11:20:56.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.1 2024-04-25T11:20:56.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.492+0000 7fd86affd640 1 -- 172.21.15.136:0/1082888822 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd87012a020 con 0x7fd86c0092d0 2024-04-25T11:20:56.490 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 msgr2=0x7fd858071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 0x7fd858071940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd86c005920 tx=0x7fd864009060 comp rx=0 tx=0).stop 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 msgr2=0x7fd86c0da700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd87001c200 tx=0x7fd87001b370 comp rx=0 tx=0).stop 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 shutdown_connections 2024-04-25T11:20:56.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd86c0dfaa0 0x7fd86c0cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd85806f430 0x7fd858071940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd86c0d6220 0x7fd86c0de560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 --2- 172.21.15.136:0/1082888822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd86c0092d0 0x7fd86c0da700 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 >> 172.21.15.136:0/1082888822 conn(0x7fd86c0514d0 msgr2=0x7fd86c0d5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 shutdown_connections 2024-04-25T11:20:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.496+0000 7fd868ff9640 1 -- 172.21.15.136:0/1082888822 wait complete. 2024-04-25T11:20:56.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-04-25T11:20:56.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.500+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f30280059d0 con 0x7f302804a1d0 2024-04-25T11:20:56.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.500+0000 7f30177fe640 1 -- 172.21.15.136:0/3375624186 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f302005ac90 con 0x7f302804a1d0 2024-04-25T11:20:56.497 INFO:teuthology.orchestra.run.smithi136.stdout:55834574859 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 msgr2=0x7f301c071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 0x7f301c071a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f302804d160 tx=0x7f303006f040 comp rx=0 tx=0).stop 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 msgr2=0x7f30280e5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f302000c3f0 tx=0x7f302000c8b0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 shutdown_connections 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f302804a1d0 0x7f30280e5d10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f30280e4440 0x7f30280488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f301c06f500 0x7f301c071a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 --2- 172.21.15.136:0/3375624186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3028006d50 0x7f3028181740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 >> 172.21.15.136:0/3375624186 conn(0x7f3028050c50 msgr2=0x7f302804eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 shutdown_connections 2024-04-25T11:20:56.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.504+0000 7f3034a10640 1 -- 172.21.15.136:0/3375624186 wait complete. 2024-04-25T11:20:56.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.5 2024-04-25T11:20:56.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.3 2024-04-25T11:20:56.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f7958050a90 con 0x7f79580064b0 2024-04-25T11:20:56.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f7955ffb640 1 -- 172.21.15.136:0/3552900620 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f795c0f61c0 con 0x7f79580064b0 2024-04-25T11:20:56.517 INFO:teuthology.orchestra.run.smithi136.stdout:55834574859 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 msgr2=0x7f7944071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 0x7f7944071910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7950009a00 tx=0x7f7950009110 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 msgr2=0x7f79580c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 0x7f79580c7940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7958051360 tx=0x7f795c01ad30 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 shutdown_connections 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79580360c0 0x7f79580df1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794406f400 0x7f7944071910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79580064b0 0x7f79580c7940 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 --2- 172.21.15.136:0/3552900620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79580060a0 0x7f795804f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 >> 172.21.15.136:0/3552900620 conn(0x7f79580503c0 msgr2=0x7f7958005790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 shutdown_connections 2024-04-25T11:20:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:56.520+0000 7f79672b3640 1 -- 172.21.15.136:0/3552900620 wait complete. 2024-04-25T11:20:56.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.0 2024-04-25T11:20:56.527 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574860 for osd.6 2024-04-25T11:20:56.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574859 for osd.7 2024-04-25T11:20:57.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T11:20:57.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T11:20:57.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T11:20:57.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T11:20:57.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T11:20:57.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T11:20:57.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T11:20:57.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T11:20:57.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.672+0000 7fdc6e624640 1 Processor -- start 2024-04-25T11:20:57.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.672+0000 7fdc6e624640 1 -- start start 2024-04-25T11:20:57.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6e624640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdc680c3010 0x7fdc6806e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6e624640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdc68157ec0 0x7fdc681582d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6e624640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc6806ea50 con 0x7fdc68152360 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6e624640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdc6806ecc0 con 0x7fdc680c3010 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6e624640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdc680c3640 con 0x7fdc68157ec0 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc67fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdc680c3010 0x7fdc6806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc677fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdc68152360 0x7fdc680af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:53138/0 (socket says 172.21.15.136:53138) 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc677fe640 1 -- 172.21.15.136:0/3407087509 learned_addr learned my addr 172.21.15.136:0/3407087509 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc6cbc1640 1 --2- 172.21.15.136:0/3407087509 >> v2:172.21.15.157:3300/0 conn(0x7fdc68157ec0 0x7fdc681582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc66ffd640 1 -- 172.21.15.136:0/3407087509 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104376384 0 0) 0x7fdc6806ea50 con 0x7fdc68152360 2024-04-25T11:20:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc66ffd640 1 -- 172.21.15.136:0/3407087509 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc50003530 con 0x7fdc68152360 2024-04-25T11:20:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc67fff640 1 -- 172.21.15.136:0/3407087509 >> v2:172.21.15.157:3300/0 conn(0x7fdc68157ec0 msgr2=0x7fdc681582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc67fff640 1 --2- 172.21.15.136:0/3407087509 >> v2:172.21.15.157:3300/0 conn(0x7fdc68157ec0 0x7fdc681582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc67fff640 1 -- 172.21.15.136:0/3407087509 >> v1:172.21.15.157:6789/0 conn(0x7fdc68152360 legacy=0x7fdc680af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.676+0000 7fdc67fff640 1 -- 172.21.15.136:0/3407087509 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc680c3920 con 0x7fdc680c3010 2024-04-25T11:20:57.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.680+0000 7fdc67fff640 1 --2- 172.21.15.136:0/3407087509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc680c3010 0x7fdc6806e510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c2240 tx=0x7fdc5c01d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4468f30d7ae8f56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.680+0000 7fdc66ffd640 1 -- 172.21.15.136:0/3407087509 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0a8070 con 0x7fdc680c3010 2024-04-25T11:20:57.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.680+0000 7fdc66ffd640 1 -- 172.21.15.136:0/3407087509 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc5c0b1070 con 0x7fdc680c3010 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 -- 172.21.15.136:0/3407087509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc680c3010 msgr2=0x7fdc6806e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc66ffd640 1 -- 172.21.15.136:0/3407087509 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0a8070 con 0x7fdc680c3010 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 --2- 172.21.15.136:0/3407087509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc680c3010 0x7fdc6806e510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c2240 tx=0x7fdc5c01d590 comp rx=0 tx=0).stop 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 -- 172.21.15.136:0/3407087509 shutdown_connections 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 --2- 172.21.15.136:0/3407087509 >> v2:172.21.15.157:3300/0 conn(0x7fdc68157ec0 0x7fdc681582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 --2- 172.21.15.136:0/3407087509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc680c3010 0x7fdc6806e510 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c2240 tx=0x7fdc5c01d590 comp rx=0 tx=0).stop 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 -- 172.21.15.136:0/3407087509 >> 172.21.15.136:0/3407087509 conn(0x7fdc680cdae0 msgr2=0x7fdc680c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 -- 172.21.15.136:0/3407087509 shutdown_connections 2024-04-25T11:20:57.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.684+0000 7fdc6e624640 1 -- 172.21.15.136:0/3407087509 wait complete. 2024-04-25T11:20:57.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.688+0000 7fdc6e624640 1 Processor -- start 2024-04-25T11:20:57.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.688+0000 7fdc6e624640 1 -- start start 2024-04-25T11:20:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.688+0000 7fdc6e624640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 0x7fdc68146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 0x7fdc68150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 0x7fdc68146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 0x7fdc68146020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35138/0 (socket says 172.21.15.136:35138) 2024-04-25T11:20:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc67fff640 1 -- 172.21.15.136:0/2018152731 learned_addr learned my addr 172.21.15.136:0/2018152731 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc680c8d00 0x7fdc68150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdc680afed0 con 0x7fdc68152360 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 0x7fdc68150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdc680afd50 con 0x7fdc68157ec0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdc68164f80 con 0x7fdc680c8d00 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6cbc1640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc680c8d00 0x7fdc68150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 -- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc680c8d00 msgr2=0x7fdc68150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc680c8d00 0x7fdc68150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 -- 172.21.15.136:0/2018152731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 msgr2=0x7fdc68146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 0x7fdc68146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc68146560 con 0x7fdc68157ec0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc677fe640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 0x7fdc68150a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000caa0 tx=0x7fdc6000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60017070 con 0x7fdc68157ec0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc68155410 con 0x7fdc68157ec0 2024-04-25T11:20:57.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc68145820 con 0x7fdc68157ec0 2024-04-25T11:20:57.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.692+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6000ac00 con 0x7fdc68157ec0 2024-04-25T11:20:57.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60012400 con 0x7fdc68157ec0 2024-04-25T11:20:57.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc480052a0 con 0x7fdc68157ec0 2024-04-25T11:20:57.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc6001a070 con 0x7fdc68157ec0 2024-04-25T11:20:57.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc64ff9640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 0x7fdc54071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc6008f200 con 0x7fdc68157ec0 2024-04-25T11:20:57.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc67fff640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 0x7fdc54071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.696+0000 7fdc67fff640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 0x7fdc54071910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c01bc40 tx=0x7fdc5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.712+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdc6005b760 con 0x7fdc68157ec0 2024-04-25T11:20:57.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.728+0000 7f136cefd640 1 Processor -- start 2024-04-25T11:20:57.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.728+0000 7f136cefd640 1 -- start start 2024-04-25T11:20:57.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136cefd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136cefd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f136815b5d0 0x7f136815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136cefd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1368070b10 con 0x7f1368070700 2024-04-25T11:20:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136cefd640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f13680c8ee0 con 0x7f136815b5d0 2024-04-25T11:20:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136cefd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f13680729c0 con 0x7f13680c82f0 2024-04-25T11:20:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45318/0 (socket says 172.21.15.136:45318) 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 -- 172.21.15.136:0/2220951802 learned_addr learned my addr 172.21.15.136:0/2220951802 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f1366d9d640 1 --2- 172.21.15.136:0/2220951802 >> v2:172.21.15.136:3300/0 conn(0x7f136815b5d0 0x7f136815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 -- 172.21.15.136:0/2220951802 >> v2:172.21.15.136:3300/0 conn(0x7f136815b5d0 msgr2=0x7f136815b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 --2- 172.21.15.136:0/2220951802 >> v2:172.21.15.136:3300/0 conn(0x7f136815b5d0 0x7f136815b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 -- 172.21.15.136:0/2220951802 >> v1:172.21.15.136:6789/0 conn(0x7f1368070700 legacy=0x7f13680c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 -- 172.21.15.136:0/2220951802 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1368073f30 con 0x7f13680c82f0 2024-04-25T11:20:57.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.732+0000 7f136659c640 1 --2- 172.21.15.136:0/2220951802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f135c013180 tx=0x7f135c024bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e5a0c09fa325254 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 Processor -- start 2024-04-25T11:20:57.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 -- start start 2024-04-25T11:20:57.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136559a640 1 -- 172.21.15.136:0/2220951802 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0112d0 con 0x7f13680c82f0 2024-04-25T11:20:57.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136559a640 1 -- 172.21.15.136:0/2220951802 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135c011470 con 0x7f13680c82f0 2024-04-25T11:20:57.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136559a640 1 -- 172.21.15.136:0/2220951802 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c00f850 con 0x7f13680c82f0 2024-04-25T11:20:57.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 -- 172.21.15.136:0/2220951802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 msgr2=0x7f13681522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 --2- 172.21.15.136:0/2220951802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f135c013180 tx=0x7f135c024bc0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd618047460 0x7fd6180d3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd61804f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6180e1800 con 0x7fd618047b20 2024-04-25T11:20:57.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd6180356a0 con 0x7fd618047460 2024-04-25T11:20:57.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd624d83640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd61804d400 con 0x7fd61804eff0 2024-04-25T11:20:57.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd618047460 0x7fd6180d3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd618047460 0x7fd6180d3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41270/0 (socket says 172.21.15.136:41270) 2024-04-25T11:20:57.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 -- 172.21.15.136:0/898015660 learned_addr learned my addr 172.21.15.136:0/898015660 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61ffff640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd61804f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 -- 172.21.15.136:0/2220951802 shutdown_connections 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 --2- 172.21.15.136:0/2220951802 >> v2:172.21.15.136:3300/0 conn(0x7f136815b5d0 0x7f136815b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 --2- 172.21.15.136:0/2220951802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f13681522f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 -- 172.21.15.136:0/2220951802 >> 172.21.15.136:0/2220951802 conn(0x7f13680c9d50 msgr2=0x7f13680ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 -- 172.21.15.136:0/2220951802 shutdown_connections 2024-04-25T11:20:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7f136cefd640 1 -- 172.21.15.136:0/2220951802 wait complete. 2024-04-25T11:20:57.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 -- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 msgr2=0x7fd61804f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd61804f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 -- 172.21.15.136:0/898015660 >> v1:172.21.15.157:6789/0 conn(0x7fd618047b20 legacy=0x7fd6180ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.736+0000 7fd61effd640 1 -- 172.21.15.136:0/898015660 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61804d6e0 con 0x7fd618047460 2024-04-25T11:20:57.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.740+0000 7f136cefd640 1 Processor -- start 2024-04-25T11:20:57.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.740+0000 7fd61effd640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd6180d3ff0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd614009870 tx=0x7fd61401c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cb06b6f31f98c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.740+0000 7f136cefd640 1 -- start start 2024-04-25T11:20:57.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f136814fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 0x7f13680cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41280/0 (socket says 172.21.15.136:41280) 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1365d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f136814fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1365d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f136814fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45342/0 (socket says 172.21.15.136:45342) 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1366d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 0x7f13680cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1366d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 0x7f13680cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35154/0 (socket says 172.21.15.136:35154) 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 -- 172.21.15.136:0/754572360 learned_addr learned my addr 172.21.15.136:0/754572360 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f136815f150 con 0x7f136815b5d0 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13680c9a80 con 0x7f1368070700 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f136815f2d0 con 0x7f13680c82f0 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 -- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 msgr2=0x7f136814fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f136814fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 -- 172.21.15.136:0/754572360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 msgr2=0x7f13680cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 0x7f13680cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13681f5810 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136659c640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f135c0b7040 tx=0x7f135c021960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0c4020 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135c0ad790 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0adb80 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1368151610 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13681476a0 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.744+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136806a350 con 0x7f1368070700 2024-04-25T11:20:57.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f135c0c7020 con 0x7f1368070700 2024-04-25T11:20:57.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd61e7fc640 1 -- 172.21.15.136:0/898015660 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b0070 con 0x7fd618047460 2024-04-25T11:20:57.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd61e7fc640 1 -- 172.21.15.136:0/898015660 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140ab600 con 0x7fd618047460 2024-04-25T11:20:57.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd61e7fc640 1 -- 172.21.15.136:0/898015660 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140ab9d0 con 0x7fd618047460 2024-04-25T11:20:57.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7f1356ffd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 0x7f1350071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f135c132170 con 0x7f1368070700 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7f1365d9b640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 0x7f1350071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 -- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 msgr2=0x7fd6180d3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd6180d3ff0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd614009870 tx=0x7fd61401c520 comp rx=0 tx=0).stop 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 -- 172.21.15.136:0/898015660 shutdown_connections 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd61804f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 --2- 172.21.15.136:0/898015660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd6180d3ff0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 -- 172.21.15.136:0/898015660 >> 172.21.15.136:0/898015660 conn(0x7fd618009fe0 msgr2=0x7fd61800a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7f1365d9b640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 0x7f1350071aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f136814b000 tx=0x7f1358009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 -- 172.21.15.136:0/898015660 shutdown_connections 2024-04-25T11:20:57.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.748+0000 7fd624d83640 1 -- 172.21.15.136:0/898015660 wait complete. 2024-04-25T11:20:57.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.752+0000 7fd624d83640 1 Processor -- start 2024-04-25T11:20:57.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.752+0000 7fd624d83640 1 -- start start 2024-04-25T11:20:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd61804ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd618052ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd61804eda0 con 0x7fd618047b20 2024-04-25T11:20:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd61804ec20 con 0x7fd618047460 2024-04-25T11:20:57.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6180489b0 con 0x7fd61804eff0 2024-04-25T11:20:57.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35166/0 (socket says 172.21.15.136:35166) 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 -- 172.21.15.136:0/762099195 learned_addr learned my addr 172.21.15.136:0/762099195 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61f7fe640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd61804ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61ffff640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd618052ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 -- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 msgr2=0x7fd618052ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd618052ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 -- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 msgr2=0x7fd61804ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd61804ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 -- 172.21.15.136:0/762099195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd618007360 con 0x7fd618047b20 2024-04-25T11:20:57.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd61effd640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd61401cdc0 tx=0x7fd6140192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b0050 con 0x7fd618047b20 2024-04-25T11:20:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140ba070 con 0x7fd618047b20 2024-04-25T11:20:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b5680 con 0x7fd618047b20 2024-04-25T11:20:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61804b0c0 con 0x7fd618047b20 2024-04-25T11:20:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6180d6c60 con 0x7fd618047b20 2024-04-25T11:20:57.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.756+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6180059d0 con 0x7fd618047b20 2024-04-25T11:20:57.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6140b5820 con 0x7fd618047b20 2024-04-25T11:20:57.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7fd60bfff640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 0x7fd604071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd614129250 con 0x7fd618047b20 2024-04-25T11:20:57.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7fd61f7fe640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 0x7fd604071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7fd61f7fe640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 0x7fd604071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0098e0 tx=0x7fd60c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.760+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f135c0ce070 con 0x7f1368070700 2024-04-25T11:20:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.768+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd614016110 con 0x7fd618047b20 2024-04-25T11:20:57.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.768+0000 7f0dd02ed640 1 Processor -- start 2024-04-25T11:20:57.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.768+0000 7f0dd02ed640 1 -- start start 2024-04-25T11:20:57.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dd02ed640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0dc8169b10 0x7f0dc816bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dd02ed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc8160270 con 0x7f0dc80c29f0 2024-04-25T11:20:57.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dd02ed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc80c92a0 con 0x7f0dc806f910 2024-04-25T11:20:57.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dd02ed640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0dc80c9820 con 0x7f0dc8169b10 2024-04-25T11:20:57.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd888640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0dc80c29f0 0x7f0dc8073b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49736/0 (socket says 172.21.15.136:49736) 2024-04-25T11:20:57.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd888640 1 -- 172.21.15.136:0/3977010176 learned_addr learned my addr 172.21.15.136:0/3977010176 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce88a640 1 --2- 172.21.15.136:0/3977010176 >> v2:172.21.15.157:3300/0 conn(0x7f0dc8169b10 0x7f0dc816bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2293114870 0 0) 0x7f0dc8160270 con 0x7f0dc80c29f0 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0db4003530 con 0x7f0dc80c29f0 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169282502 0 0) 0x7f0dc80c92a0 con 0x7f0dc806f910 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc8160270 con 0x7f0dc806f910 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1208114536 0 0) 0x7f0db4003530 con 0x7f0dc80c29f0 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 >> v2:172.21.15.157:3300/0 conn(0x7f0dc8169b10 msgr2=0x7f0dc816bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 --2- 172.21.15.136:0/3977010176 >> v2:172.21.15.157:3300/0 conn(0x7f0dc8169b10 0x7f0dc816bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 >> v1:172.21.15.157:6789/0 conn(0x7f0dc806f910 legacy=0x7f0dc80c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc806ef20 con 0x7f0dc80c29f0 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f0db8001680 con 0x7f0dc80c29f0 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 >> v1:172.21.15.136:6789/0 conn(0x7f0dc80c29f0 legacy=0x7f0dc8073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 0x7f0db4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4004810 0x7f0db4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0db4005280 0x7f0db400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80c9820 con 0x7f0db4003b50 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0db400c210 con 0x7f0db4005280 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0db400c410 con 0x7f0db4004810 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 0x7f0db4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dcd888640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4004810 0x7f0db4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 -- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4004810 msgr2=0x7f0db4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4004810 0x7f0db4004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 -- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0db4005280 msgr2=0x7f0db400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce88a640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0db4005280 0x7f0db400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0db4005280 0x7f0db400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce089640 1 -- 172.21.15.136:0/3977010176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db4003530 con 0x7f0db4003b50 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce88a640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0db4005280 0x7f0db400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.772+0000 7f0dce88a640 1 -- 172.21.15.136:0/3977010176 reap_dead start 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 Processor -- start 2024-04-25T11:20:57.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 -- start start 2024-04-25T11:20:57.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0dce089640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 0x7f0db4003f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4001950 tx=0x7f0dc401c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac144e6021fe92b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40b1070 con 0x7f0db4003b50 2024-04-25T11:20:57.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0374152120 0x7f03740c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc40ac630 con 0x7f0db4003b50 2024-04-25T11:20:57.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0dcd087640 1 -- 172.21.15.136:0/3977010176 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40a8050 con 0x7f0db4003b50 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0372d9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0374072c90 0x7f0374158b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:53158/0 (socket says 172.21.15.136:53158) 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0372d9d640 1 -- 172.21.15.136:0/2019129578 learned_addr learned my addr 172.21.15.136:0/2019129578 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f037259c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0374152120 0x7f03740c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03740730a0 con 0x7f03740703c0 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0374152530 con 0x7f0374072c90 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f03740c7ea0 con 0x7f0374152120 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0371d9b640 1 -- 172.21.15.136:0/2019129578 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827589030 0 0) 0x7f0374152530 con 0x7f0374072c90 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f0371d9b640 1 -- 172.21.15.136:0/2019129578 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f035c003530 con 0x7f0374072c90 2024-04-25T11:20:57.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f037259c640 1 -- 172.21.15.136:0/2019129578 >> v1:172.21.15.157:6789/0 conn(0x7f0374072c90 legacy=0x7f0374158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f037259c640 1 -- 172.21.15.136:0/2019129578 >> v1:172.21.15.136:6789/0 conn(0x7f03740703c0 legacy=0x7f0374162a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.776+0000 7f037259c640 1 -- 172.21.15.136:0/2019129578 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03740c80f0 con 0x7f0374152120 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 msgr2=0x7f0db4003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 0x7f0db4003f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4001950 tx=0x7f0dc401c5a0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- 172.21.15.136:0/3977010176 shutdown_connections 2024-04-25T11:20:57.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/3977010176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0db4003b50 0x7f0db4003f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- 172.21.15.136:0/3977010176 >> 172.21.15.136:0/3977010176 conn(0x7f0dc80cdae0 msgr2=0x7f0dc80c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- 172.21.15.136:0/3977010176 shutdown_connections 2024-04-25T11:20:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7fdd870fb640 1 Processor -- start 2024-04-25T11:20:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7fdd870fb640 1 -- start start 2024-04-25T11:20:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7ff8520a4640 1 Processor -- start 2024-04-25T11:20:57.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7ff8520a4640 1 -- start start 2024-04-25T11:20:57.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- 172.21.15.136:0/3977010176 wait complete. 2024-04-25T11:20:57.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 Processor -- start 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f037259c640 1 --2- 172.21.15.136:0/2019129578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374152120 0x7f03740c9190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0368009a20 tx=0x7f036801d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83d39e896588779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0371d9b640 1 -- 172.21.15.136:0/2019129578 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680a7670 con 0x7f0374152120 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0371d9b640 1 -- 172.21.15.136:0/2019129578 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03680a7810 con 0x7f0374152120 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd870fb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdd8006e7a0 0x7fdd800c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd870fb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdd800c9ef0 0x7fdd80166800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd870fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd80156640 con 0x7fdd80156230 2024-04-25T11:20:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd870fb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdd800c8a60 con 0x7fdd800c9ef0 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd870fb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8015f680 con 0x7fdd8006e7a0 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.780+0000 7f0dd02ed640 1 -- start start 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd860f9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fdd80156230 0x7fdd8015fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49746/0 (socket says 172.21.15.136:49746) 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd860f9640 1 -- 172.21.15.136:0/70103133 learned_addr learned my addr 172.21.15.136:0/70103133 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd868fa640 1 --2- 172.21.15.136:0/70103133 >> v2:172.21.15.136:3300/0 conn(0x7fdd800c9ef0 0x7fdd80166800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 0x7f0dc8151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0dc81fcc50 0x7f0dc815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff84c0c2e00 0x7ff84c15bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49758/0 (socket says 172.21.15.136:49758) 2024-04-25T11:20:57.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84b7fe640 1 -- 172.21.15.136:0/1640797783 learned_addr learned my addr 172.21.15.136:0/1640797783 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45354/0 (socket says 172.21.15.136:45354) 2024-04-25T11:20:57.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dcd888640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 0x7f0dc8151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dcd888640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 0x7f0dc8151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41330/0 (socket says 172.21.15.136:41330) 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd850f7640 1 -- 172.21.15.136:0/70103133 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458749749 0 0) 0x7fdd80156640 con 0x7fdd80156230 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd850f7640 1 -- 172.21.15.136:0/70103133 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd6c003530 con 0x7fdd80156230 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd868fa640 1 -- 172.21.15.136:0/70103133 >> v2:172.21.15.157:3300/0 conn(0x7fdd8006e7a0 msgr2=0x7fdd800c99b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd868fa640 1 --2- 172.21.15.136:0/70103133 >> v2:172.21.15.157:3300/0 conn(0x7fdd8006e7a0 0x7fdd800c99b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd868fa640 1 -- 172.21.15.136:0/70103133 >> v1:172.21.15.136:6789/0 conn(0x7fdd80156230 legacy=0x7fdd8015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7fdd868fa640 1 -- 172.21.15.136:0/70103133 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd800c8c90 con 0x7fdd800c9ef0 2024-04-25T11:20:57.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc8075150 con 0x7f0dc81fcc50 2024-04-25T11:20:57.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 -- 172.21.15.136:0/161761286 learned_addr learned my addr 172.21.15.136:0/161761286 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff8520a4640 1 --2- 172.21.15.136:0/1640797783 >> v2:172.21.15.136:3300/0 conn(0x7ff84c0749a0 0x7ff84c074db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff8520a4640 1 --2- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce88a640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0dc81fcc50 0x7f0dc815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc8074fd0 con 0x7f0dc8156a90 2024-04-25T11:20:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c0c3210 con 0x7ff84c0c2e00 2024-04-25T11:20:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff84c15cae0 con 0x7ff84c0749a0 2024-04-25T11:20:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84affd640 1 --2- 172.21.15.136:0/1640797783 >> v2:172.21.15.136:3300/0 conn(0x7ff84c0749a0 0x7ff84c074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0371d9b640 1 -- 172.21.15.136:0/2019129578 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680a7670 con 0x7f0374152120 2024-04-25T11:20:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff84c0c9d20 con 0x7ff84c0c4340 2024-04-25T11:20:57.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374152120 msgr2=0x7f03740c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 --2- 172.21.15.136:0/2019129578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374152120 0x7f03740c9190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0368009a20 tx=0x7f036801d7d0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc8073b20 con 0x7f0db4003b50 2024-04-25T11:20:57.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84bfff640 1 --2- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 shutdown_connections 2024-04-25T11:20:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 --2- 172.21.15.136:0/2019129578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374152120 0x7f03740c9190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 >> 172.21.15.136:0/2019129578 conn(0x7f03740cd4a0 msgr2=0x7f03740cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 shutdown_connections 2024-04-25T11:20:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84bfff640 1 -- 172.21.15.136:0/1640797783 >> v2:172.21.15.136:3300/0 conn(0x7ff84c0749a0 msgr2=0x7ff84c074db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84bfff640 1 --2- 172.21.15.136:0/1640797783 >> v2:172.21.15.136:3300/0 conn(0x7ff84c0749a0 0x7ff84c074db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84bfff640 1 -- 172.21.15.136:0/1640797783 >> v1:172.21.15.136:6789/0 conn(0x7ff84c0c2e00 legacy=0x7ff84c15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7ff84bfff640 1 -- 172.21.15.136:0/1640797783 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c0c4c90 con 0x7ff84c0c4340 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 -- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 msgr2=0x7f0dc8151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 0x7f0dc8151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 -- 172.21.15.136:0/161761286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0dc81fcc50 msgr2=0x7f0dc815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0dc81fcc50 0x7f0dc815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc815ede0 con 0x7f0db4003b50 2024-04-25T11:20:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f0dce089640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0dc401c3d0 tx=0x7f0db4005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.784+0000 7f03791de640 1 -- 172.21.15.136:0/2019129578 wait complete. 2024-04-25T11:20:57.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 Processor -- start 2024-04-25T11:20:57.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 -- start start 2024-04-25T11:20:57.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd868fa640 1 --2- 172.21.15.136:0/70103133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd800c9ef0 0x7fdd80166800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd800c42f0 tx=0x7fdd7801cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a04416f12eea8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd850f7640 1 -- 172.21.15.136:0/70103133 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780b2070 con 0x7fdd800c9ef0 2024-04-25T11:20:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 0x7f0374141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0374152120 0x7f037413f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd850f7640 1 -- 172.21.15.136:0/70103133 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd780a5b90 con 0x7fdd800c9ef0 2024-04-25T11:20:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd850f7640 1 -- 172.21.15.136:0/70103133 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780ad440 con 0x7fdd800c9ef0 2024-04-25T11:20:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35216/0 (socket says 172.21.15.136:35216) 2024-04-25T11:20:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0372d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 0x7f0374141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0374070be0 con 0x7f0374072c90 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03740c96d0 con 0x7f0374152120 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03740c9c80 con 0x7f03740703c0 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0372d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 0x7f0374141cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45362/0 (socket says 172.21.15.136:45362) 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0372d9d640 1 -- 172.21.15.136:0/1631522718 learned_addr learned my addr 172.21.15.136:0/1631522718 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff84bfff640 1 --2- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c0c4750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff844009c80 tx=0x7ff84401d960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c08b27c15d9496c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40b9070 con 0x7f0db4003b50 2024-04-25T11:20:57.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 -- 172.21.15.136:0/70103133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd800c9ef0 msgr2=0x7fdd80166800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 --2- 172.21.15.136:0/70103133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd800c9ef0 0x7fdd80166800 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdd800c42f0 tx=0x7fdd7801cbb0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037359e640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0374152120 0x7f037413f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff84a7fc640 1 -- 172.21.15.136:0/1640797783 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8440b3070 con 0x7ff84c0c4340 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff84a7fc640 1 -- 172.21.15.136:0/1640797783 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8440a67a0 con 0x7ff84c0c4340 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff84a7fc640 1 -- 172.21.15.136:0/1640797783 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8440a6b70 con 0x7ff84c0c4340 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 -- 172.21.15.136:0/70103133 shutdown_connections 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 --2- 172.21.15.136:0/70103133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd800c9ef0 0x7fdd80166800 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 --2- 172.21.15.136:0/70103133 >> v2:172.21.15.157:3300/0 conn(0x7fdd8006e7a0 0x7fdd800c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 -- 172.21.15.136:0/70103133 >> 172.21.15.136:0/70103133 conn(0x7fdd8006d100 msgr2=0x7fdd80070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc8155dd0 con 0x7f0db4003b50 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc8156390 con 0x7f0db4003b50 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc40b5410 con 0x7f0db4003b50 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40b1040 con 0x7f0db4003b50 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 -- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 msgr2=0x7f0374141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 0x7f0374141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 -- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0374152120 msgr2=0x7f037413f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0374152120 0x7f037413f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 -- 172.21.15.136:0/1631522718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0374143320 con 0x7f0374072c90 2024-04-25T11:20:57.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff02d89e640 1 Processor -- start 2024-04-25T11:20:57.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f037259c640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0374073270 tx=0x7f036801a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368004040 con 0x7f0374072c90 2024-04-25T11:20:57.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 msgr2=0x7ff84c0c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 --2- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c0c4750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff844009c80 tx=0x7ff84401d960 comp rx=0 tx=0).stop 2024-04-25T11:20:57.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 shutdown_connections 2024-04-25T11:20:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 --2- 172.21.15.136:0/1640797783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c0c4750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 --2- 172.21.15.136:0/1640797783 >> v2:172.21.15.136:3300/0 conn(0x7ff84c0749a0 0x7ff84c074db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 >> 172.21.15.136:0/1640797783 conn(0x7ff84c0ca5a0 msgr2=0x7ff84c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 -- 172.21.15.136:0/70103133 shutdown_connections 2024-04-25T11:20:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0368004200 con 0x7f0374072c90 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368003620 con 0x7f0374072c90 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7ff02d89e640 1 -- start start 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff02d89e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff0281522f0 0x7ff028070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff02d89e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 0x7ff02815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff02d89e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff028152700 con 0x7ff0280c9370 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff02d89e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff0280c8ee0 con 0x7ff0280c9780 2024-04-25T11:20:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff02d89e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff0280c9b90 con 0x7ff0281522f0 2024-04-25T11:20:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7fdd870fb640 1 -- 172.21.15.136:0/70103133 wait complete. 2024-04-25T11:20:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff0267fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff0280c9370 0x7ff02815b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:53170/0 (socket says 172.21.15.136:53170) 2024-04-25T11:20:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff0267fc640 1 -- 172.21.15.136:0/3905798369 learned_addr learned my addr 172.21.15.136:0/3905798369 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff026ffd640 1 --2- 172.21.15.136:0/3905798369 >> v2:172.21.15.157:3300/0 conn(0x7ff0281522f0 0x7ff028070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.788+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03741435e0 con 0x7f0374072c90 2024-04-25T11:20:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f03680037c0 con 0x7f0374072c90 2024-04-25T11:20:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0dc40bc070 con 0x7f0db4003b50 2024-04-25T11:20:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0dbeffd640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 0x7f0dac071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0dc4128fc0 con 0x7f0db4003b50 2024-04-25T11:20:57.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff0277fe640 1 --2- 172.21.15.136:0/3905798369 >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 0x7ff02815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff025ffb640 1 -- 172.21.15.136:0/3905798369 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77272447 0 0) 0x7ff028152700 con 0x7ff0280c9370 2024-04-25T11:20:57.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0dcd888640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 0x7f0dac071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f035b7fe640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 0x7f0360071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff025ffb640 1 -- 172.21.15.136:0/3905798369 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff010003530 con 0x7ff0280c9370 2024-04-25T11:20:57.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff026ffd640 1 -- 172.21.15.136:0/3905798369 >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 msgr2=0x7ff02815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741f4c40 con 0x7f0374072c90 2024-04-25T11:20:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0372d9d640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 0x7f0360071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f03680a7770 con 0x7f0374072c90 2024-04-25T11:20:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 shutdown_connections 2024-04-25T11:20:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff8520a4640 1 -- 172.21.15.136:0/1640797783 wait complete. 2024-04-25T11:20:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0dcd888640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 0x7f0dac071a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0dc81f4020 tx=0x7f0db8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7f0372d9d640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 0x7f0360071990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f03640011f0 tx=0x7f0364009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 Processor -- start 2024-04-25T11:20:57.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 -- start start 2024-04-25T11:20:57.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff026ffd640 1 --2- 172.21.15.136:0/3905798369 >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 0x7ff02815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff026ffd640 1 -- 172.21.15.136:0/3905798369 >> v1:172.21.15.157:6789/0 conn(0x7ff0280c9370 legacy=0x7ff02815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7ff026ffd640 1 -- 172.21.15.136:0/3905798369 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0280729c0 con 0x7ff0281522f0 2024-04-25T11:20:57.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff0277fe640 1 --2- 172.21.15.136:0/3905798369 >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 0x7ff02815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:20:57.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7fdd870fb640 1 Processor -- start 2024-04-25T11:20:57.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.792+0000 7fdd870fb640 1 -- start start 2024-04-25T11:20:57.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 0x7fdd801f8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 0x7fdd8015c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45372/0 (socket says 172.21.15.136:45372) 2024-04-25T11:20:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd858f8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 0x7fdd801f8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd858f8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 0x7fdd801f8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41374/0 (socket says 172.21.15.136:41374) 2024-04-25T11:20:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd868fa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 0x7fdd8015c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd868fa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 0x7fdd8015c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35226/0 (socket says 172.21.15.136:35226) 2024-04-25T11:20:57.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 -- 172.21.15.136:0/1329978847 learned_addr learned my addr 172.21.15.136:0/1329978847 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8006ebb0 con 0x7fdd80156230 2024-04-25T11:20:57.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd800c2020 con 0x7fdd800c9ef0 2024-04-25T11:20:57.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd80160220 con 0x7fdd8006e7a0 2024-04-25T11:20:57.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff026ffd640 1 --2- 172.21.15.136:0/3905798369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028070700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff028071ee0 tx=0x7ff01801e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fadd3a91b8f4da5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 -- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 msgr2=0x7fdd801f8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 0x7fdd801f8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 -- 172.21.15.136:0/1329978847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 msgr2=0x7fdd8015c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 0x7fdd8015c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd800cbf00 con 0x7fdd8006e7a0 2024-04-25T11:20:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff025ffb640 1 -- 172.21.15.136:0/3905798369 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180ad070 con 0x7ff0281522f0 2024-04-25T11:20:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7fdd860f9640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000c420 tx=0x7fdd7000c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff025ffb640 1 -- 172.21.15.136:0/3905798369 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0180a7970 con 0x7ff0281522f0 2024-04-25T11:20:57.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff025ffb640 1 -- 172.21.15.136:0/3905798369 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180a7d20 con 0x7ff0281522f0 2024-04-25T11:20:57.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 -- 172.21.15.136:0/3905798369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 msgr2=0x7ff028070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 --2- 172.21.15.136:0/3905798369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028070700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff028071ee0 tx=0x7ff01801e9d0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 -- 172.21.15.136:0/3905798369 shutdown_connections 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 --2- 172.21.15.136:0/3905798369 >> v2:172.21.15.136:3300/0 conn(0x7ff0280c9780 0x7ff02815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 --2- 172.21.15.136:0/3905798369 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028070700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff02d89e640 1 -- 172.21.15.136:0/3905798369 >> 172.21.15.136:0/3905798369 conn(0x7ff0280c9d50 msgr2=0x7ff0280ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff02d89e640 1 -- 172.21.15.136:0/3905798369 shutdown_connections 2024-04-25T11:20:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff84c0749a0 0x7ff84c1f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c1532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c15c4d0 con 0x7ff84c0c2e00 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c0ca000 con 0x7ff84c0749a0 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff8520a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff84c073200 con 0x7ff84c0c4340 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff84affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff84affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35242/0 (socket says 172.21.15.136:35242) 2024-04-25T11:20:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff84affd640 1 -- 172.21.15.136:0/4156212884 learned_addr learned my addr 172.21.15.136:0/4156212884 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff84bfff640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c1532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7ff84b7fe640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff84c0749a0 0x7ff84c1f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 -- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 msgr2=0x7ff84c1532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c1532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 -- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff84c0749a0 msgr2=0x7ff84c1f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff84c0749a0 0x7ff84c1f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 -- 172.21.15.136:0/4156212884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c147420 con 0x7ff84c0c2e00 2024-04-25T11:20:57.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.796+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03741f4a60 con 0x7f0374072c90 2024-04-25T11:20:57.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff84affd640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00e870 tx=0x7ff83c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff02d89e640 1 -- 172.21.15.136:0/3905798369 wait complete. 2024-04-25T11:20:57.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff02d89e640 1 Processor -- start 2024-04-25T11:20:57.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70017070 con 0x7fdd8006e7a0 2024-04-25T11:20:57.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db00052a0 con 0x7f0db4003b50 2024-04-25T11:20:57.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7ff02d89e640 1 -- start start 2024-04-25T11:20:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd70005cf0 con 0x7fdd8006e7a0 2024-04-25T11:20:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.800+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8015f1e0 con 0x7fdd8006e7a0 2024-04-25T11:20:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f03741f4a60 con 0x7f0374072c90 2024-04-25T11:20:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd700072d0 con 0x7fdd8006e7a0 2024-04-25T11:20:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd80202de0 con 0x7fdd8006e7a0 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 0x7ff028156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45380/0 (socket says 172.21.15.136:45380) 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 -- 172.21.15.136:0/1317296428 learned_addr learned my addr 172.21.15.136:0/1317296428 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff0267fc640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 0x7ff028156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff02815c3a0 con 0x7ff0280c9780 2024-04-25T11:20:57.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8006a2e0 con 0x7fdd8006e7a0 2024-04-25T11:20:57.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff02815c220 con 0x7ff0281522f0 2024-04-25T11:20:57.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff028165180 con 0x7ff0280c9370 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83c017070 con 0x7ff84c0c2e00 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff83c00ee60 con 0x7ff84c0c2e00 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff83c00cb40 con 0x7ff84c0c2e00 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff8520a4640 1 -- 172.21.15.136:0/4156212884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84c147700 con 0x7ff84c0c2e00 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 -- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 msgr2=0x7ff028154c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:20:57.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff8520a4640 1 -- 172.21.15.136:0/4156212884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff84c1f7100 con 0x7ff84c0c2e00 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028154c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 -- 172.21.15.136:0/1317296428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 msgr2=0x7ff028156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 0x7ff028156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff028157510 con 0x7ff0280c9370 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.804+0000 7ff026ffd640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff01801c600 tx=0x7ff01801b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff83c01a070 con 0x7ff84c0c2e00 2024-04-25T11:20:57.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff0267fc640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 0x7ff028156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8240052a0 con 0x7ff84c0c2e00 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff833fff640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 0x7ff838071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff83c08f820 con 0x7ff84c0c2e00 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff84b7fe640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 0x7ff838071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.808+0000 7ff84b7fe640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 0x7ff838071910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff840005890 tx=0x7ff840007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180c0070 con 0x7ff0280c9370 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0dc40162b0 con 0x7f0db4003b50 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff018008cb0 con 0x7ff0280c9370 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180ad050 con 0x7ff0280c9370 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff028168e90 con 0x7ff0280c9370 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0281567d0 con 0x7ff0280c9370 2024-04-25T11:20:57.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff02806a350 con 0x7ff0280c9370 2024-04-25T11:20:57.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd7001a070 con 0x7fdd8006e7a0 2024-04-25T11:20:57.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.812+0000 7fdd76ffd640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 0x7fdd640719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.816+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdd7008ee50 con 0x7fdd8006e7a0 2024-04-25T11:20:57.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.816+0000 7fdd858f8640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 0x7fdd640719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.816+0000 7fdd858f8640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 0x7fdd640719e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd8015eef0 tx=0x7fdd7c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.820+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff83c05bd00 con 0x7ff84c0c2e00 2024-04-25T11:20:57.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.820+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdd7005b390 con 0x7fdd8006e7a0 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.824+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff018004240 con 0x7ff0280c9370 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.824+0000 7ff00ffff640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 0x7ff014071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.824+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff018129640 con 0x7ff0280c9370 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.824+0000 7ff0267fc640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 0x7ff014071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.824+0000 7ff0267fc640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 0x7ff014071940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff02814ef10 tx=0x7ff020009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:57.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.832+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff0180ca020 con 0x7ff0280c9370 2024-04-25T11:20:57.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.996+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdc48005860 con 0x7fdc68157ec0 2024-04-25T11:20:57.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:57.996+0000 7fdc64ff9640 1 -- 172.21.15.136:0/2018152731 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc6005fa60 con 0x7fdc68157ec0 2024-04-25T11:20:57.998 INFO:teuthology.orchestra.run.smithi136.stdout:55834574863 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 msgr2=0x7fdc54071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 0x7fdc54071910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c01bc40 tx=0x7fdc5c008040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 msgr2=0x7fdc68150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 0x7fdc68150a50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000caa0 tx=0x7fdc6000cf60 comp rx=0 tx=0).stop 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 shutdown_connections 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc5406f400 0x7fdc54071910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdc680c8d00 0x7fdc68150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc68157ec0 0x7fdc68150a50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.000+0000 7fdc6e624640 1 --2- 172.21.15.136:0/2018152731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc68152360 0x7fdc68146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.004+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 >> 172.21.15.136:0/2018152731 conn(0x7fdc680cdae0 msgr2=0x7fdc680cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.004+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 shutdown_connections 2024-04-25T11:20:58.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.004+0000 7fdc6e624640 1 -- 172.21.15.136:0/2018152731 wait complete. 2024-04-25T11:20:58.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574863 for osd.4 2024-04-25T11:20:58.027 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.080+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1368155a00 con 0x7f1368070700 2024-04-25T11:20:58.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.080+0000 7f1356ffd640 1 -- 172.21.15.136:0/754572360 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f135c01e840 con 0x7f1368070700 2024-04-25T11:20:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 msgr2=0x7f1350071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 0x7f1350071aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f136814b000 tx=0x7f1358009040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 msgr2=0x7f136815c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f135c0b7040 tx=0x7f135c021960 comp rx=0 tx=0).stop 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 shutdown_connections 2024-04-25T11:20:58.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f136815b5d0 0x7f13680cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f135006f590 0x7f1350071aa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f13680c82f0 0x7f136814fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 --2- 172.21.15.136:0/754572360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1368070700 0x7f136815c750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 >> 172.21.15.136:0/754572360 conn(0x7f13680c9d50 msgr2=0x7f13681612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 shutdown_connections 2024-04-25T11:20:58.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.084+0000 7f136cefd640 1 -- 172.21.15.136:0/754572360 wait complete. 2024-04-25T11:20:58.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.096+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd61804e7f0 con 0x7fd618047b20 2024-04-25T11:20:58.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.096+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff02806f340 con 0x7ff0280c9370 2024-04-25T11:20:58.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.096+0000 7fd60bfff640 1 -- 172.21.15.136:0/762099195 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd6140f5e90 con 0x7fd618047b20 2024-04-25T11:20:58.093 INFO:teuthology.orchestra.run.smithi136.stdout:55834574860 2024-04-25T11:20:58.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.096+0000 7ff00ffff640 1 -- 172.21.15.136:0/1317296428 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff0180185c0 con 0x7ff0280c9370 2024-04-25T11:20:58.094 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:58.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 msgr2=0x7fd604071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 msgr2=0x7ff014071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 0x7ff014071940 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff02814ef10 tx=0x7ff020009040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 msgr2=0x7ff0281fa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff01801c600 tx=0x7ff01801b770 comp rx=0 tx=0).stop 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 shutdown_connections 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff0281522f0 0x7ff028154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0280c9780 0x7ff028156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff01406f430 0x7ff014071940 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 --2- 172.21.15.136:0/1317296428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff0280c9370 0x7ff0281fa730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 >> 172.21.15.136:0/1317296428 conn(0x7ff0280c9d50 msgr2=0x7ff0280cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 0x7fd604071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0098e0 tx=0x7fd60c009060 comp rx=0 tx=0).stop 2024-04-25T11:20:58.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 shutdown_connections 2024-04-25T11:20:58.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 msgr2=0x7fd6180dfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7ff02d89e640 1 -- 172.21.15.136:0/1317296428 wait complete. 2024-04-25T11:20:58.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd61401cdc0 tx=0x7fd6140192c0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 shutdown_connections 2024-04-25T11:20:58.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd60406f430 0x7fd604071940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd61804eff0 0x7fd618052ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd618047b20 0x7fd6180dfa80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 --2- 172.21.15.136:0/762099195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd618047460 0x7fd61804ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 >> 172.21.15.136:0/762099195 conn(0x7fd618009fe0 msgr2=0x7fd618005c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 shutdown_connections 2024-04-25T11:20:58.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.100+0000 7fd624d83640 1 -- 172.21.15.136:0/762099195 wait complete. 2024-04-25T11:20:58.105 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574861 for osd.1 2024-04-25T11:20:58.105 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.116+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0db0005590 con 0x7f0db4003b50 2024-04-25T11:20:58.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.116+0000 7f0dbeffd640 1 -- 172.21.15.136:0/161761286 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0dc40f5590 con 0x7f0db4003b50 2024-04-25T11:20:58.116 INFO:teuthology.orchestra.run.smithi136.stdout:55834574862 2024-04-25T11:20:58.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 msgr2=0x7f0dac071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 0x7f0dac071a10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0dc81f4020 tx=0x7f0db8009040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 msgr2=0x7f0dc80c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0dc401c3d0 tx=0x7f0db4005840 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 shutdown_connections 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0dc81fcc50 0x7f0dc815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc8156a90 0x7f0dc8151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0dac06f500 0x7f0dac071a10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 --2- 172.21.15.136:0/161761286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0db4003b50 0x7f0dc80c7140 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 >> 172.21.15.136:0/161761286 conn(0x7f0dc80cdae0 msgr2=0x7f0dc815d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 shutdown_connections 2024-04-25T11:20:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.120+0000 7f0dd02ed640 1 -- 172.21.15.136:0/161761286 wait complete. 2024-04-25T11:20:58.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.124+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f037406a350 con 0x7f0374072c90 2024-04-25T11:20:58.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.124+0000 7f035b7fe640 1 -- 172.21.15.136:0/1631522718 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f03680ad030 con 0x7f0374072c90 2024-04-25T11:20:58.121 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 msgr2=0x7f0360071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 0x7f0360071990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f03640011f0 tx=0x7f0364009040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 msgr2=0x7f03741f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0374073270 tx=0x7f036801a570 comp rx=0 tx=0).stop 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 shutdown_connections 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0374152120 0x7f037413f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0374072c90 0x7f03741f5440 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f036006f480 0x7f0360071990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 --2- 172.21.15.136:0/1631522718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03740703c0 0x7f0374141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 >> 172.21.15.136:0/1631522718 conn(0x7f03740cd4a0 msgr2=0x7f03740c5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 shutdown_connections 2024-04-25T11:20:58.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.128+0000 7f03791de640 1 -- 172.21.15.136:0/1631522718 wait complete. 2024-04-25T11:20:58.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574861 for osd.3 2024-04-25T11:20:58.127 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574860 for osd.2 2024-04-25T11:20:58.128 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574862 for osd.5 2024-04-25T11:20:58.149 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.7 2024-04-25T11:20:58.151 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdd800c40d0 con 0x7fdd8006e7a0 2024-04-25T11:20:58.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd76ffd640 1 -- 172.21.15.136:0/1329978847 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd7005f690 con 0x7fdd8006e7a0 2024-04-25T11:20:58.153 INFO:teuthology.orchestra.run.smithi136.stdout:55834574862 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 msgr2=0x7fdd640719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 0x7fdd640719e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd8015eef0 tx=0x7fdd7c009060 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 msgr2=0x7fdd800cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000c420 tx=0x7fdd7000c8e0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 shutdown_connections 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd80156230 0x7fdd8015c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd800c9ef0 0x7fdd801f8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd6406f4d0 0x7fdd640719e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 --2- 172.21.15.136:0/1329978847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd8006e7a0 0x7fdd800cb9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 >> 172.21.15.136:0/1329978847 conn(0x7fdd8006d100 msgr2=0x7fdd8006ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 shutdown_connections 2024-04-25T11:20:58.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.156+0000 7fdd870fb640 1 -- 172.21.15.136:0/1329978847 wait complete. 2024-04-25T11:20:58.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff824005860 con 0x7ff84c0c2e00 2024-04-25T11:20:58.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff833fff640 1 -- 172.21.15.136:0/4156212884 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff83c060000 con 0x7ff84c0c2e00 2024-04-25T11:20:58.158 INFO:teuthology.orchestra.run.smithi136.stdout:55834574861 2024-04-25T11:20:58.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 msgr2=0x7ff838071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 0x7ff838071910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff840005890 tx=0x7ff840007cb0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 msgr2=0x7ff84c14a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00e870 tx=0x7ff83c00ed30 comp rx=0 tx=0).stop 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 shutdown_connections 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff83806f400 0x7ff838071910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff84c0c4340 0x7ff84c1532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff84c0c2e00 0x7ff84c14a170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 --2- 172.21.15.136:0/4156212884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff84c0749a0 0x7ff84c1f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 >> 172.21.15.136:0/4156212884 conn(0x7ff84c0ca5a0 msgr2=0x7ff84c0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 shutdown_connections 2024-04-25T11:20:58.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.160+0000 7ff831ffb640 1 -- 172.21.15.136:0/4156212884 wait complete. 2024-04-25T11:20:58.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574862 got 55834574862 for osd.6 2024-04-25T11:20:58.175 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574861 for osd.0 2024-04-25T11:20:58.178 DEBUG:teuthology.parallel:result is None 2024-04-25T11:20:58.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T11:20:58.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:20:58.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 Processor -- start 2024-04-25T11:20:58.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 -- start start 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb764156750 0x7fb7640ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7640ca7d0 0x7fb7640cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb76415c410 con 0x7fb7640cf9d0 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb7640cfde0 con 0x7fb7640ca7d0 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7694ae640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb76415c6b0 con 0x7fb764156750 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7640ca7d0 0x7fb7640cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7640ca7d0 0x7fb7640cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35248/0 (socket says 172.21.15.136:35248) 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 -- 172.21.15.136:0/2759210934 learned_addr learned my addr 172.21.15.136:0/2759210934 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb762ffd640 1 --2- 172.21.15.136:0/2759210934 >> v2:172.21.15.157:3300/0 conn(0x7fb764156750 0x7fb7640ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 -- 172.21.15.136:0/2759210934 >> v2:172.21.15.157:3300/0 conn(0x7fb764156750 msgr2=0x7fb7640ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 --2- 172.21.15.136:0/2759210934 >> v2:172.21.15.157:3300/0 conn(0x7fb764156750 0x7fb7640ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 -- 172.21.15.136:0/2759210934 >> v1:172.21.15.157:6789/0 conn(0x7fb7640cf9d0 legacy=0x7fb76406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:58.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.352+0000 7fb7627fc640 1 -- 172.21.15.136:0/2759210934 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76415c990 con 0x7fb7640ca7d0 2024-04-25T11:20:58.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7627fc640 1 --2- 172.21.15.136:0/2759210934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640cf490 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7640d00d0 tx=0x7fb75801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53061c44206e969 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb761ffb640 1 -- 172.21.15.136:0/2759210934 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580b0070 con 0x7fb7640ca7d0 2024-04-25T11:20:58.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb761ffb640 1 -- 172.21.15.136:0/2759210934 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7580ab600 con 0x7fb7640ca7d0 2024-04-25T11:20:58.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb761ffb640 1 -- 172.21.15.136:0/2759210934 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580ab9d0 con 0x7fb7640ca7d0 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 -- 172.21.15.136:0/2759210934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 msgr2=0x7fb7640cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 --2- 172.21.15.136:0/2759210934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640cf490 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7640d00d0 tx=0x7fb75801c4e0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 -- 172.21.15.136:0/2759210934 shutdown_connections 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 --2- 172.21.15.136:0/2759210934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640cf490 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 --2- 172.21.15.136:0/2759210934 >> v2:172.21.15.157:3300/0 conn(0x7fb764156750 0x7fb7640ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 -- 172.21.15.136:0/2759210934 >> 172.21.15.136:0/2759210934 conn(0x7fb764070f70 msgr2=0x7fb764068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 -- 172.21.15.136:0/2759210934 shutdown_connections 2024-04-25T11:20:58.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.356+0000 7fb7694ae640 1 -- 172.21.15.136:0/2759210934 wait complete. 2024-04-25T11:20:58.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 Processor -- start 2024-04-25T11:20:58.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- start start 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7640cf9d0 0x7fb764149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb76414eda0 0x7fb76414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb764169440 con 0x7fb7640ca7d0 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb7641692c0 con 0x7fb7640cf9d0 2024-04-25T11:20:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7641695c0 con 0x7fb76414eda0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35264/0 (socket says 172.21.15.136:35264) 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 -- 172.21.15.136:0/314240149 learned_addr learned my addr 172.21.15.136:0/314240149 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7637fe640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb76414eda0 0x7fb76414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7627fc640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7640cf9d0 0x7fb764149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 -- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb76414eda0 msgr2=0x7fb76414f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb76414eda0 0x7fb76414f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 -- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7640cf9d0 msgr2=0x7fb764149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7640cf9d0 0x7fb764149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 -- 172.21.15.136:0/314240149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7640ce880 con 0x7fb7640ca7d0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb762ffd640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007e40 tx=0x7fb75c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c014620 con 0x7fb7640ca7d0 2024-04-25T11:20:58.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb75c019070 con 0x7fb7640ca7d0 2024-04-25T11:20:58.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76414a270 con 0x7fb7640ca7d0 2024-04-25T11:20:58.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c014a00 con 0x7fb7640ca7d0 2024-04-25T11:20:58.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76414fb90 con 0x7fb7640ca7d0 2024-04-25T11:20:58.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76406a350 con 0x7fb7640ca7d0 2024-04-25T11:20:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.360+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb75c014ba0 con 0x7fb7640ca7d0 2024-04-25T11:20:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.364+0000 7fb74b7fe640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 0x7fb750071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.364+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb75c08fe90 con 0x7fb7640ca7d0 2024-04-25T11:20:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.364+0000 7fb7627fc640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 0x7fb750071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.364+0000 7fb7627fc640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 0x7fb750071910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7580ad000 tx=0x7fb758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.368+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb7640ce880 con 0x7fb7640ca7d0 2024-04-25T11:20:58.460 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:20:58.462+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:20:58.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.512+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb764150d30 con 0x7fb75006f400 2024-04-25T11:20:58.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.536+0000 7fb74b7fe640 1 -- 172.21.15.136:0/314240149 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332526 (secure 0 0 0) 0x7fb764150d30 con 0x7fb75006f400 2024-04-25T11:20:58.537 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:58.539 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:20:58.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 msgr2=0x7fb750071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 0x7fb750071910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7580ad000 tx=0x7fb758008040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 msgr2=0x7fb7640b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb75c007e40 tx=0x7fb75c00e4b0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 shutdown_connections 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75006f400 0x7fb750071910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb76414eda0 0x7fb76414f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7640cf9d0 0x7fb764149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 --2- 172.21.15.136:0/314240149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7640ca7d0 0x7fb7640b4fd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 >> 172.21.15.136:0/314240149 conn(0x7fb764070f70 msgr2=0x7fb764072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 shutdown_connections 2024-04-25T11:20:58.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.540+0000 7fb7694ae640 1 -- 172.21.15.136:0/314240149 wait complete. 2024-04-25T11:20:58.557 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-25T11:20:56.766988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":150,"num_write_kb":1647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236000,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738720,"statfs":{"total":773094113280,"available":772852449280,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005239"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070611+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:25.070611+0000","last_peered":"2024-04-25T11:20:25.070611+0000","last_clean":"2024-04-25T11:20:25.070611+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:25.070611+0000","last_undegraded":"2024-04-25T11:20:25.070611+0000","last_fullsized":"2024-04-25T11:20:25.070611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066703+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:25.066703+0000","last_peered":"2024-04-25T11:20:25.066703+0000","last_clean":"2024-04-25T11:20:25.066703+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:25.066703+0000","last_undegraded":"2024-04-25T11:20:25.066703+0000","last_fullsized":"2024-04-25T11:20:25.066703+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070831+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:25.070831+0000","last_peered":"2024-04-25T11:20:25.070831+0000","last_clean":"2024-04-25T11:20:25.070831+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:25.070831+0000","last_undegraded":"2024-04-25T11:20:25.070831+0000","last_fullsized":"2024-04-25T11:20:25.070831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848527+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:25.848527+0000","last_peered":"2024-04-25T11:20:25.848527+0000","last_clean":"2024-04-25T11:20:25.848527+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:25.848527+0000","last_undegraded":"2024-04-25T11:20:25.848527+0000","last_fullsized":"2024-04-25T11:20:25.848527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850005+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:20:25.850005+0000","last_peered":"2024-04-25T11:20:25.850005+0000","last_clean":"2024-04-25T11:20:25.850005+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:20:25.850005+0000","last_undegraded":"2024-04-25T11:20:25.850005+0000","last_fullsized":"2024-04-25T11:20:25.850005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852853+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:20:25.852853+0000","last_peered":"2024-04-25T11:20:25.852853+0000","last_clean":"2024-04-25T11:20:25.852853+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.852853+0000","last_undegraded":"2024-04-25T11:20:25.852853+0000","last_fullsized":"2024-04-25T11:20:25.852853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703772+0000","last_change":"2024-04-25T11:20:20.032982+0000","last_active":"2024-04-25T11:20:25.703772+0000","last_peered":"2024-04-25T11:20:25.703772+0000","last_clean":"2024-04-25T11:20:25.703772+0000","last_became_active":"2024-04-25T11:20:20.032127+0000","last_became_peered":"2024-04-25T11:20:20.032127+0000","last_unstale":"2024-04-25T11:20:25.703772+0000","last_undegraded":"2024-04-25T11:20:25.703772+0000","last_fullsized":"2024-04-25T11:20:25.703772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852476+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:20:25.852476+0000","last_peered":"2024-04-25T11:20:25.852476+0000","last_clean":"2024-04-25T11:20:25.852476+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:20:25.852476+0000","last_undegraded":"2024-04-25T11:20:25.852476+0000","last_fullsized":"2024-04-25T11:20:25.852476+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851378+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:20:25.851378+0000","last_peered":"2024-04-25T11:20:25.851378+0000","last_clean":"2024-04-25T11:20:25.851378+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:20:25.851378+0000","last_undegraded":"2024-04-25T11:20:25.851378+0000","last_fullsized":"2024-04-25T11:20:25.851378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704317+0000","last_change":"2024-04-25T11:20:20.031174+0000","last_active":"2024-04-25T11:20:25.704317+0000","last_peered":"2024-04-25T11:20:25.704317+0000","last_clean":"2024-04-25T11:20:25.704317+0000","last_became_active":"2024-04-25T11:20:20.030340+0000","last_became_peered":"2024-04-25T11:20:20.030340+0000","last_unstale":"2024-04-25T11:20:25.704317+0000","last_undegraded":"2024-04-25T11:20:25.704317+0000","last_fullsized":"2024-04-25T11:20:25.704317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850069+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:20:25.850069+0000","last_peered":"2024-04-25T11:20:25.850069+0000","last_clean":"2024-04-25T11:20:25.850069+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:20:25.850069+0000","last_undegraded":"2024-04-25T11:20:25.850069+0000","last_fullsized":"2024-04-25T11:20:25.850069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065389+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:25.065389+0000","last_peered":"2024-04-25T11:20:25.065389+0000","last_clean":"2024-04-25T11:20:25.065389+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:25.065389+0000","last_undegraded":"2024-04-25T11:20:25.065389+0000","last_fullsized":"2024-04-25T11:20:25.065389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850811+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:20:25.850811+0000","last_peered":"2024-04-25T11:20:25.850811+0000","last_clean":"2024-04-25T11:20:25.850811+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:20:25.850811+0000","last_undegraded":"2024-04-25T11:20:25.850811+0000","last_fullsized":"2024-04-25T11:20:25.850811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851666+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:20:25.851666+0000","last_peered":"2024-04-25T11:20:25.851666+0000","last_clean":"2024-04-25T11:20:25.851666+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:20:25.851666+0000","last_undegraded":"2024-04-25T11:20:25.851666+0000","last_fullsized":"2024-04-25T11:20:25.851666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067265+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:25.067265+0000","last_peered":"2024-04-25T11:20:25.067265+0000","last_clean":"2024-04-25T11:20:25.067265+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:25.067265+0000","last_undegraded":"2024-04-25T11:20:25.067265+0000","last_fullsized":"2024-04-25T11:20:25.067265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850338+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:20:25.850338+0000","last_peered":"2024-04-25T11:20:25.850338+0000","last_clean":"2024-04-25T11:20:25.850338+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:20:25.850338+0000","last_undegraded":"2024-04-25T11:20:25.850338+0000","last_fullsized":"2024-04-25T11:20:25.850338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850575+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:20:25.850575+0000","last_peered":"2024-04-25T11:20:25.850575+0000","last_clean":"2024-04-25T11:20:25.850575+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:20:25.850575+0000","last_undegraded":"2024-04-25T11:20:25.850575+0000","last_fullsized":"2024-04-25T11:20:25.850575+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066984+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:25.066984+0000","last_peered":"2024-04-25T11:20:25.066984+0000","last_clean":"2024-04-25T11:20:25.066984+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:25.066984+0000","last_undegraded":"2024-04-25T11:20:25.066984+0000","last_fullsized":"2024-04-25T11:20:25.066984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848318+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:20:25.848318+0000","last_peered":"2024-04-25T11:20:25.848318+0000","last_clean":"2024-04-25T11:20:25.848318+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:20:25.848318+0000","last_undegraded":"2024-04-25T11:20:25.848318+0000","last_fullsized":"2024-04-25T11:20:25.848318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850089+0000","last_change":"2024-04-25T11:20:24.165156+0000","last_active":"2024-04-25T11:20:25.850089+0000","last_peered":"2024-04-25T11:20:25.850089+0000","last_clean":"2024-04-25T11:20:25.850089+0000","last_became_active":"2024-04-25T11:20:24.164587+0000","last_became_peered":"2024-04-25T11:20:24.164587+0000","last_unstale":"2024-04-25T11:20:25.850089+0000","last_undegraded":"2024-04-25T11:20:25.850089+0000","last_fullsized":"2024-04-25T11:20:25.850089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851436+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:20:25.851436+0000","last_peered":"2024-04-25T11:20:25.851436+0000","last_clean":"2024-04-25T11:20:25.851436+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:20:25.851436+0000","last_undegraded":"2024-04-25T11:20:25.851436+0000","last_fullsized":"2024-04-25T11:20:25.851436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849318+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:20:25.849318+0000","last_peered":"2024-04-25T11:20:25.849318+0000","last_clean":"2024-04-25T11:20:25.849318+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:20:25.849318+0000","last_undegraded":"2024-04-25T11:20:25.849318+0000","last_fullsized":"2024-04-25T11:20:25.849318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850973+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:20:25.850973+0000","last_peered":"2024-04-25T11:20:25.850973+0000","last_clean":"2024-04-25T11:20:25.850973+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:20:25.850973+0000","last_undegraded":"2024-04-25T11:20:25.850973+0000","last_fullsized":"2024-04-25T11:20:25.850973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849979+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:25.849979+0000","last_peered":"2024-04-25T11:20:25.849979+0000","last_clean":"2024-04-25T11:20:25.849979+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:25.849979+0000","last_undegraded":"2024-04-25T11:20:25.849979+0000","last_fullsized":"2024-04-25T11:20:25.849979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066481+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:20:25.066481+0000","last_peered":"2024-04-25T11:20:25.066481+0000","last_clean":"2024-04-25T11:20:25.066481+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:20:25.066481+0000","last_undegraded":"2024-04-25T11:20:25.066481+0000","last_fullsized":"2024-04-25T11:20:25.066481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850564+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:20:25.850564+0000","last_peered":"2024-04-25T11:20:25.850564+0000","last_clean":"2024-04-25T11:20:25.850564+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:20:25.850564+0000","last_undegraded":"2024-04-25T11:20:25.850564+0000","last_fullsized":"2024-04-25T11:20:25.850564+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071084+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:25.071084+0000","last_peered":"2024-04-25T11:20:25.071084+0000","last_clean":"2024-04-25T11:20:25.071084+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:25.071084+0000","last_undegraded":"2024-04-25T11:20:25.071084+0000","last_fullsized":"2024-04-25T11:20:25.071084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850091+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:20:25.850091+0000","last_peered":"2024-04-25T11:20:25.850091+0000","last_clean":"2024-04-25T11:20:25.850091+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:20:25.850091+0000","last_undegraded":"2024-04-25T11:20:25.850091+0000","last_fullsized":"2024-04-25T11:20:25.850091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066330+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:25.066330+0000","last_peered":"2024-04-25T11:20:25.066330+0000","last_clean":"2024-04-25T11:20:25.066330+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:25.066330+0000","last_undegraded":"2024-04-25T11:20:25.066330+0000","last_fullsized":"2024-04-25T11:20:25.066330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850885+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:25.850885+0000","last_peered":"2024-04-25T11:20:25.850885+0000","last_clean":"2024-04-25T11:20:25.850885+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:25.850885+0000","last_undegraded":"2024-04-25T11:20:25.850885+0000","last_fullsized":"2024-04-25T11:20:25.850885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853257+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:20:25.853257+0000","last_peered":"2024-04-25T11:20:25.853257+0000","last_clean":"2024-04-25T11:20:25.853257+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.853257+0000","last_undegraded":"2024-04-25T11:20:25.853257+0000","last_fullsized":"2024-04-25T11:20:25.853257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848798+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:25.848798+0000","last_peered":"2024-04-25T11:20:25.848798+0000","last_clean":"2024-04-25T11:20:25.848798+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:25.848798+0000","last_undegraded":"2024-04-25T11:20:25.848798+0000","last_fullsized":"2024-04-25T11:20:25.848798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704541+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.704541+0000","last_peered":"2024-04-25T11:20:25.704541+0000","last_clean":"2024-04-25T11:20:25.704541+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:20:25.704541+0000","last_undegraded":"2024-04-25T11:20:25.704541+0000","last_fullsized":"2024-04-25T11:20:25.704541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850957+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:25.850957+0000","last_peered":"2024-04-25T11:20:25.850957+0000","last_clean":"2024-04-25T11:20:25.850957+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:25.850957+0000","last_undegraded":"2024-04-25T11:20:25.850957+0000","last_fullsized":"2024-04-25T11:20:25.850957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848973+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:25.848973+0000","last_peered":"2024-04-25T11:20:25.848973+0000","last_clean":"2024-04-25T11:20:25.848973+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:25.848973+0000","last_undegraded":"2024-04-25T11:20:25.848973+0000","last_fullsized":"2024-04-25T11:20:25.848973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852994+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:20:25.852994+0000","last_peered":"2024-04-25T11:20:25.852994+0000","last_clean":"2024-04-25T11:20:25.852994+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:20:25.852994+0000","last_undegraded":"2024-04-25T11:20:25.852994+0000","last_fullsized":"2024-04-25T11:20:25.852994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703782+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:20:25.703782+0000","last_peered":"2024-04-25T11:20:25.703782+0000","last_clean":"2024-04-25T11:20:25.703782+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:20:25.703782+0000","last_undegraded":"2024-04-25T11:20:25.703782+0000","last_fullsized":"2024-04-25T11:20:25.703782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850807+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:25.850807+0000","last_peered":"2024-04-25T11:20:25.850807+0000","last_clean":"2024-04-25T11:20:25.850807+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:25.850807+0000","last_undegraded":"2024-04-25T11:20:25.850807+0000","last_fullsized":"2024-04-25T11:20:25.850807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850780+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:25.850780+0000","last_peered":"2024-04-25T11:20:25.850780+0000","last_clean":"2024-04-25T11:20:25.850780+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:25.850780+0000","last_undegraded":"2024-04-25T11:20:25.850780+0000","last_fullsized":"2024-04-25T11:20:25.850780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066095+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:20:25.066095+0000","last_peered":"2024-04-25T11:20:25.066095+0000","last_clean":"2024-04-25T11:20:25.066095+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:20:25.066095+0000","last_undegraded":"2024-04-25T11:20:25.066095+0000","last_fullsized":"2024-04-25T11:20:25.066095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850239+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:25.850239+0000","last_peered":"2024-04-25T11:20:25.850239+0000","last_clean":"2024-04-25T11:20:25.850239+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:25.850239+0000","last_undegraded":"2024-04-25T11:20:25.850239+0000","last_fullsized":"2024-04-25T11:20:25.850239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848175+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:25.848175+0000","last_peered":"2024-04-25T11:20:25.848175+0000","last_clean":"2024-04-25T11:20:25.848175+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:25.848175+0000","last_undegraded":"2024-04-25T11:20:25.848175+0000","last_fullsized":"2024-04-25T11:20:25.848175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849105+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:20:25.849105+0000","last_peered":"2024-04-25T11:20:25.849105+0000","last_clean":"2024-04-25T11:20:25.849105+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:20:25.849105+0000","last_undegraded":"2024-04-25T11:20:25.849105+0000","last_fullsized":"2024-04-25T11:20:25.849105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849839+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:25.849839+0000","last_peered":"2024-04-25T11:20:25.849839+0000","last_clean":"2024-04-25T11:20:25.849839+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:25.849839+0000","last_undegraded":"2024-04-25T11:20:25.849839+0000","last_fullsized":"2024-04-25T11:20:25.849839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066978+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:25.066978+0000","last_peered":"2024-04-25T11:20:25.066978+0000","last_clean":"2024-04-25T11:20:25.066978+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:25.066978+0000","last_undegraded":"2024-04-25T11:20:25.066978+0000","last_fullsized":"2024-04-25T11:20:25.066978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703462+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:20:25.703462+0000","last_peered":"2024-04-25T11:20:25.703462+0000","last_clean":"2024-04-25T11:20:25.703462+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:20:25.703462+0000","last_undegraded":"2024-04-25T11:20:25.703462+0000","last_fullsized":"2024-04-25T11:20:25.703462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849435+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:20:25.849435+0000","last_peered":"2024-04-25T11:20:25.849435+0000","last_clean":"2024-04-25T11:20:25.849435+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:20:25.849435+0000","last_undegraded":"2024-04-25T11:20:25.849435+0000","last_fullsized":"2024-04-25T11:20:25.849435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066131+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:25.066131+0000","last_peered":"2024-04-25T11:20:25.066131+0000","last_clean":"2024-04-25T11:20:25.066131+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:25.066131+0000","last_undegraded":"2024-04-25T11:20:25.066131+0000","last_fullsized":"2024-04-25T11:20:25.066131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852310+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:20:25.852310+0000","last_peered":"2024-04-25T11:20:25.852310+0000","last_clean":"2024-04-25T11:20:25.852310+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:20:25.852310+0000","last_undegraded":"2024-04-25T11:20:25.852310+0000","last_fullsized":"2024-04-25T11:20:25.852310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065502+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:25.065502+0000","last_peered":"2024-04-25T11:20:25.065502+0000","last_clean":"2024-04-25T11:20:25.065502+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:25.065502+0000","last_undegraded":"2024-04-25T11:20:25.065502+0000","last_fullsized":"2024-04-25T11:20:25.065502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703910+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:20:25.703910+0000","last_peered":"2024-04-25T11:20:25.703910+0000","last_clean":"2024-04-25T11:20:25.703910+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:20:25.703910+0000","last_undegraded":"2024-04-25T11:20:25.703910+0000","last_fullsized":"2024-04-25T11:20:25.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849639+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:20:25.849639+0000","last_peered":"2024-04-25T11:20:25.849639+0000","last_clean":"2024-04-25T11:20:25.849639+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:20:25.849639+0000","last_undegraded":"2024-04-25T11:20:25.849639+0000","last_fullsized":"2024-04-25T11:20:25.849639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849150+0000","last_change":"2024-04-25T11:20:24.098651+0000","last_active":"2024-04-25T11:20:25.849150+0000","last_peered":"2024-04-25T11:20:25.849150+0000","last_clean":"2024-04-25T11:20:25.849150+0000","last_became_active":"2024-04-25T11:20:24.097985+0000","last_became_peered":"2024-04-25T11:20:24.097985+0000","last_unstale":"2024-04-25T11:20:25.849150+0000","last_undegraded":"2024-04-25T11:20:25.849150+0000","last_fullsized":"2024-04-25T11:20:25.849150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852724+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:20:25.852724+0000","last_peered":"2024-04-25T11:20:25.852724+0000","last_clean":"2024-04-25T11:20:25.852724+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:20:25.852724+0000","last_undegraded":"2024-04-25T11:20:25.852724+0000","last_fullsized":"2024-04-25T11:20:25.852724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067792+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:20:25.067792+0000","last_peered":"2024-04-25T11:20:25.067792+0000","last_clean":"2024-04-25T11:20:25.067792+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:20:25.067792+0000","last_undegraded":"2024-04-25T11:20:25.067792+0000","last_fullsized":"2024-04-25T11:20:25.067792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702944+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:20:25.702944+0000","last_peered":"2024-04-25T11:20:25.702944+0000","last_clean":"2024-04-25T11:20:25.702944+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:20:25.702944+0000","last_undegraded":"2024-04-25T11:20:25.702944+0000","last_fullsized":"2024-04-25T11:20:25.702944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852444+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:20:25.852444+0000","last_peered":"2024-04-25T11:20:25.852444+0000","last_clean":"2024-04-25T11:20:25.852444+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:20:25.852444+0000","last_undegraded":"2024-04-25T11:20:25.852444+0000","last_fullsized":"2024-04-25T11:20:25.852444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851726+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:20:25.851726+0000","last_peered":"2024-04-25T11:20:25.851726+0000","last_clean":"2024-04-25T11:20:25.851726+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:20:25.851726+0000","last_undegraded":"2024-04-25T11:20:25.851726+0000","last_fullsized":"2024-04-25T11:20:25.851726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850209+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:20:25.850209+0000","last_peered":"2024-04-25T11:20:25.850209+0000","last_clean":"2024-04-25T11:20:25.850209+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:20:25.850209+0000","last_undegraded":"2024-04-25T11:20:25.850209+0000","last_fullsized":"2024-04-25T11:20:25.850209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849636+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:20:25.849636+0000","last_peered":"2024-04-25T11:20:25.849636+0000","last_clean":"2024-04-25T11:20:25.849636+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:20:25.849636+0000","last_undegraded":"2024-04-25T11:20:25.849636+0000","last_fullsized":"2024-04-25T11:20:25.849636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065250+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:20:25.065250+0000","last_peered":"2024-04-25T11:20:25.065250+0000","last_clean":"2024-04-25T11:20:25.065250+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:20:25.065250+0000","last_undegraded":"2024-04-25T11:20:25.065250+0000","last_fullsized":"2024-04-25T11:20:25.065250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849363+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:25.849363+0000","last_peered":"2024-04-25T11:20:25.849363+0000","last_clean":"2024-04-25T11:20:25.849363+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:25.849363+0000","last_undegraded":"2024-04-25T11:20:25.849363+0000","last_fullsized":"2024-04-25T11:20:25.849363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065441+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:25.065441+0000","last_peered":"2024-04-25T11:20:25.065441+0000","last_clean":"2024-04-25T11:20:25.065441+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:25.065441+0000","last_undegraded":"2024-04-25T11:20:25.065441+0000","last_fullsized":"2024-04-25T11:20:25.065441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065600+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:20:25.065600+0000","last_peered":"2024-04-25T11:20:25.065600+0000","last_clean":"2024-04-25T11:20:25.065600+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:20:25.065600+0000","last_undegraded":"2024-04-25T11:20:25.065600+0000","last_fullsized":"2024-04-25T11:20:25.065600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071194+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:25.071194+0000","last_peered":"2024-04-25T11:20:25.071194+0000","last_clean":"2024-04-25T11:20:25.071194+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:25.071194+0000","last_undegraded":"2024-04-25T11:20:25.071194+0000","last_fullsized":"2024-04-25T11:20:25.071194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850312+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:20:25.850312+0000","last_peered":"2024-04-25T11:20:25.850312+0000","last_clean":"2024-04-25T11:20:25.850312+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:20:25.850312+0000","last_undegraded":"2024-04-25T11:20:25.850312+0000","last_fullsized":"2024-04-25T11:20:25.850312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703560+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:20:25.703560+0000","last_peered":"2024-04-25T11:20:25.703560+0000","last_clean":"2024-04-25T11:20:25.703560+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:20:25.703560+0000","last_undegraded":"2024-04-25T11:20:25.703560+0000","last_fullsized":"2024-04-25T11:20:25.703560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067907+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:25.067907+0000","last_peered":"2024-04-25T11:20:25.067907+0000","last_clean":"2024-04-25T11:20:25.067907+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:25.067907+0000","last_undegraded":"2024-04-25T11:20:25.067907+0000","last_fullsized":"2024-04-25T11:20:25.067907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849479+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:25.849479+0000","last_peered":"2024-04-25T11:20:25.849479+0000","last_clean":"2024-04-25T11:20:25.849479+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:25.849479+0000","last_undegraded":"2024-04-25T11:20:25.849479+0000","last_fullsized":"2024-04-25T11:20:25.849479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849577+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:20:25.849577+0000","last_peered":"2024-04-25T11:20:25.849577+0000","last_clean":"2024-04-25T11:20:25.849577+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:20:25.849577+0000","last_undegraded":"2024-04-25T11:20:25.849577+0000","last_fullsized":"2024-04-25T11:20:25.849577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704488+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:20:25.704488+0000","last_peered":"2024-04-25T11:20:25.704488+0000","last_clean":"2024-04-25T11:20:25.704488+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:20:25.704488+0000","last_undegraded":"2024-04-25T11:20:25.704488+0000","last_fullsized":"2024-04-25T11:20:25.704488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851990+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:20:25.851990+0000","last_peered":"2024-04-25T11:20:25.851990+0000","last_clean":"2024-04-25T11:20:25.851990+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:20:25.851990+0000","last_undegraded":"2024-04-25T11:20:25.851990+0000","last_fullsized":"2024-04-25T11:20:25.851990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:41.996974+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:20:41.996974+0000","last_peered":"2024-04-25T11:20:41.996974+0000","last_clean":"2024-04-25T11:20:41.996974+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:20:41.996974+0000","last_undegraded":"2024-04-25T11:20:41.996974+0000","last_fullsized":"2024-04-25T11:20:41.996974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849162+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:25.849162+0000","last_peered":"2024-04-25T11:20:25.849162+0000","last_clean":"2024-04-25T11:20:25.849162+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:25.849162+0000","last_undegraded":"2024-04-25T11:20:25.849162+0000","last_fullsized":"2024-04-25T11:20:25.849162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851639+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:20:25.851639+0000","last_peered":"2024-04-25T11:20:25.851639+0000","last_clean":"2024-04-25T11:20:25.851639+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:20:25.851639+0000","last_undegraded":"2024-04-25T11:20:25.851639+0000","last_fullsized":"2024-04-25T11:20:25.851639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849514+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:25.849514+0000","last_peered":"2024-04-25T11:20:25.849514+0000","last_clean":"2024-04-25T11:20:25.849514+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:25.849514+0000","last_undegraded":"2024-04-25T11:20:25.849514+0000","last_fullsized":"2024-04-25T11:20:25.849514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852817+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:20:25.852817+0000","last_peered":"2024-04-25T11:20:25.852817+0000","last_clean":"2024-04-25T11:20:25.852817+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:20:25.852817+0000","last_undegraded":"2024-04-25T11:20:25.852817+0000","last_fullsized":"2024-04-25T11:20:25.852817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848647+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:25.848647+0000","last_peered":"2024-04-25T11:20:25.848647+0000","last_clean":"2024-04-25T11:20:25.848647+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:25.848647+0000","last_undegraded":"2024-04-25T11:20:25.848647+0000","last_fullsized":"2024-04-25T11:20:25.848647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704293+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:20:25.704293+0000","last_peered":"2024-04-25T11:20:25.704293+0000","last_clean":"2024-04-25T11:20:25.704293+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:20:25.704293+0000","last_undegraded":"2024-04-25T11:20:25.704293+0000","last_fullsized":"2024-04-25T11:20:25.704293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066858+0000","last_change":"2024-04-25T11:20:24.093677+0000","last_active":"2024-04-25T11:20:25.066858+0000","last_peered":"2024-04-25T11:20:25.066858+0000","last_clean":"2024-04-25T11:20:25.066858+0000","last_became_active":"2024-04-25T11:20:24.092907+0000","last_became_peered":"2024-04-25T11:20:24.092907+0000","last_unstale":"2024-04-25T11:20:25.066858+0000","last_undegraded":"2024-04-25T11:20:25.066858+0000","last_fullsized":"2024-04-25T11:20:25.066858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:30.768495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704129+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:20:25.704129+0000","last_peered":"2024-04-25T11:20:25.704129+0000","last_clean":"2024-04-25T11:20:25.704129+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:20:25.704129+0000","last_undegraded":"2024-04-25T11:20:25.704129+0000","last_fullsized":"2024-04-25T11:20:25.704129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066326+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:20:25.066326+0000","last_peered":"2024-04-25T11:20:25.066326+0000","last_clean":"2024-04-25T11:20:25.066326+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:20:25.066326+0000","last_undegraded":"2024-04-25T11:20:25.066326+0000","last_fullsized":"2024-04-25T11:20:25.066326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067316+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:25.067316+0000","last_peered":"2024-04-25T11:20:25.067316+0000","last_clean":"2024-04-25T11:20:25.067316+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:25.067316+0000","last_undegraded":"2024-04-25T11:20:25.067316+0000","last_fullsized":"2024-04-25T11:20:25.067316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848888+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:25.848888+0000","last_peered":"2024-04-25T11:20:25.848888+0000","last_clean":"2024-04-25T11:20:25.848888+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:25.848888+0000","last_undegraded":"2024-04-25T11:20:25.848888+0000","last_fullsized":"2024-04-25T11:20:25.848888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848177+0000","last_change":"2024-04-25T11:20:20.040078+0000","last_active":"2024-04-25T11:20:25.848177+0000","last_peered":"2024-04-25T11:20:25.848177+0000","last_clean":"2024-04-25T11:20:25.848177+0000","last_became_active":"2024-04-25T11:20:20.039255+0000","last_became_peered":"2024-04-25T11:20:20.039255+0000","last_unstale":"2024-04-25T11:20:25.848177+0000","last_undegraded":"2024-04-25T11:20:25.848177+0000","last_fullsized":"2024-04-25T11:20:25.848177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703263+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:20:25.703263+0000","last_peered":"2024-04-25T11:20:25.703263+0000","last_clean":"2024-04-25T11:20:25.703263+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:20:25.703263+0000","last_undegraded":"2024-04-25T11:20:25.703263+0000","last_fullsized":"2024-04-25T11:20:25.703263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066138+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:25.066138+0000","last_peered":"2024-04-25T11:20:25.066138+0000","last_clean":"2024-04-25T11:20:25.066138+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:25.066138+0000","last_undegraded":"2024-04-25T11:20:25.066138+0000","last_fullsized":"2024-04-25T11:20:25.066138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066710+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:25.066710+0000","last_peered":"2024-04-25T11:20:25.066710+0000","last_clean":"2024-04-25T11:20:25.066710+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:25.066710+0000","last_undegraded":"2024-04-25T11:20:25.066710+0000","last_fullsized":"2024-04-25T11:20:25.066710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067819+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:20:25.067819+0000","last_peered":"2024-04-25T11:20:25.067819+0000","last_clean":"2024-04-25T11:20:25.067819+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:20:25.067819+0000","last_undegraded":"2024-04-25T11:20:25.067819+0000","last_fullsized":"2024-04-25T11:20:25.067819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065961+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:20:25.065961+0000","last_peered":"2024-04-25T11:20:25.065961+0000","last_clean":"2024-04-25T11:20:25.065961+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:20:25.065961+0000","last_undegraded":"2024-04-25T11:20:25.065961+0000","last_fullsized":"2024-04-25T11:20:25.065961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849428+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:25.849428+0000","last_peered":"2024-04-25T11:20:25.849428+0000","last_clean":"2024-04-25T11:20:25.849428+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:25.849428+0000","last_undegraded":"2024-04-25T11:20:25.849428+0000","last_fullsized":"2024-04-25T11:20:25.849428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066149+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:25.066149+0000","last_peered":"2024-04-25T11:20:25.066149+0000","last_clean":"2024-04-25T11:20:25.066149+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:25.066149+0000","last_undegraded":"2024-04-25T11:20:25.066149+0000","last_fullsized":"2024-04-25T11:20:25.066149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067406+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:25.067406+0000","last_peered":"2024-04-25T11:20:25.067406+0000","last_clean":"2024-04-25T11:20:25.067406+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:25.067406+0000","last_undegraded":"2024-04-25T11:20:25.067406+0000","last_fullsized":"2024-04-25T11:20:25.067406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850698+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:20:25.850698+0000","last_peered":"2024-04-25T11:20:25.850698+0000","last_clean":"2024-04-25T11:20:25.850698+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:20:25.850698+0000","last_undegraded":"2024-04-25T11:20:25.850698+0000","last_fullsized":"2024-04-25T11:20:25.850698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070455+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:25.070455+0000","last_peered":"2024-04-25T11:20:25.070455+0000","last_clean":"2024-04-25T11:20:25.070455+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:25.070455+0000","last_undegraded":"2024-04-25T11:20:25.070455+0000","last_fullsized":"2024-04-25T11:20:25.070455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065240+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:25.065240+0000","last_peered":"2024-04-25T11:20:25.065240+0000","last_clean":"2024-04-25T11:20:25.065240+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:25.065240+0000","last_undegraded":"2024-04-25T11:20:25.065240+0000","last_fullsized":"2024-04-25T11:20:25.065240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065744+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:25.065744+0000","last_peered":"2024-04-25T11:20:25.065744+0000","last_clean":"2024-04-25T11:20:25.065744+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:25.065744+0000","last_undegraded":"2024-04-25T11:20:25.065744+0000","last_fullsized":"2024-04-25T11:20:25.065744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703495+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:20:25.703495+0000","last_peered":"2024-04-25T11:20:25.703495+0000","last_clean":"2024-04-25T11:20:25.703495+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:20:25.703495+0000","last_undegraded":"2024-04-25T11:20:25.703495+0000","last_fullsized":"2024-04-25T11:20:25.703495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849293+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:25.849293+0000","last_peered":"2024-04-25T11:20:25.849293+0000","last_clean":"2024-04-25T11:20:25.849293+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:25.849293+0000","last_undegraded":"2024-04-25T11:20:25.849293+0000","last_fullsized":"2024-04-25T11:20:25.849293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.068272+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:25.068272+0000","last_peered":"2024-04-25T11:20:25.068272+0000","last_clean":"2024-04-25T11:20:25.068272+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:25.068272+0000","last_undegraded":"2024-04-25T11:20:25.068272+0000","last_fullsized":"2024-04-25T11:20:25.068272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849313+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:25.849313+0000","last_peered":"2024-04-25T11:20:25.849313+0000","last_clean":"2024-04-25T11:20:25.849313+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849313+0000","last_undegraded":"2024-04-25T11:20:25.849313+0000","last_fullsized":"2024-04-25T11:20:25.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066579+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:25.066579+0000","last_peered":"2024-04-25T11:20:25.066579+0000","last_clean":"2024-04-25T11:20:25.066579+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:25.066579+0000","last_undegraded":"2024-04-25T11:20:25.066579+0000","last_fullsized":"2024-04-25T11:20:25.066579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070896+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:25.070896+0000","last_peered":"2024-04-25T11:20:25.070896+0000","last_clean":"2024-04-25T11:20:25.070896+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:25.070896+0000","last_undegraded":"2024-04-25T11:20:25.070896+0000","last_fullsized":"2024-04-25T11:20:25.070896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703122+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:20:25.703122+0000","last_peered":"2024-04-25T11:20:25.703122+0000","last_clean":"2024-04-25T11:20:25.703122+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:20:25.703122+0000","last_undegraded":"2024-04-25T11:20:25.703122+0000","last_fullsized":"2024-04-25T11:20:25.703122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067675+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:25.067675+0000","last_peered":"2024-04-25T11:20:25.067675+0000","last_clean":"2024-04-25T11:20:25.067675+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:25.067675+0000","last_undegraded":"2024-04-25T11:20:25.067675+0000","last_fullsized":"2024-04-25T11:20:25.067675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703259+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:20:25.703259+0000","last_peered":"2024-04-25T11:20:25.703259+0000","last_clean":"2024-04-25T11:20:25.703259+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:20:25.703259+0000","last_undegraded":"2024-04-25T11:20:25.703259+0000","last_fullsized":"2024-04-25T11:20:25.703259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848929+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:25.848929+0000","last_peered":"2024-04-25T11:20:25.848929+0000","last_clean":"2024-04-25T11:20:25.848929+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:25.848929+0000","last_undegraded":"2024-04-25T11:20:25.848929+0000","last_fullsized":"2024-04-25T11:20:25.848929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066246+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:25.066246+0000","last_peered":"2024-04-25T11:20:25.066246+0000","last_clean":"2024-04-25T11:20:25.066246+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:25.066246+0000","last_undegraded":"2024-04-25T11:20:25.066246+0000","last_fullsized":"2024-04-25T11:20:25.066246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853167+0000","last_change":"2024-04-25T11:20:20.039980+0000","last_active":"2024-04-25T11:20:25.853167+0000","last_peered":"2024-04-25T11:20:25.853167+0000","last_clean":"2024-04-25T11:20:25.853167+0000","last_became_active":"2024-04-25T11:20:20.039297+0000","last_became_peered":"2024-04-25T11:20:20.039297+0000","last_unstale":"2024-04-25T11:20:25.853167+0000","last_undegraded":"2024-04-25T11:20:25.853167+0000","last_fullsized":"2024-04-25T11:20:25.853167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:18.780733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067502+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:25.067502+0000","last_peered":"2024-04-25T11:20:25.067502+0000","last_clean":"2024-04-25T11:20:25.067502+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:25.067502+0000","last_undegraded":"2024-04-25T11:20:25.067502+0000","last_fullsized":"2024-04-25T11:20:25.067502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704854+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:20:25.704854+0000","last_peered":"2024-04-25T11:20:25.704854+0000","last_clean":"2024-04-25T11:20:25.704854+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:20:25.704854+0000","last_undegraded":"2024-04-25T11:20:25.704854+0000","last_fullsized":"2024-04-25T11:20:25.704854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702747+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.702747+0000","last_peered":"2024-04-25T11:20:25.702747+0000","last_clean":"2024-04-25T11:20:25.702747+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:20:25.702747+0000","last_undegraded":"2024-04-25T11:20:25.702747+0000","last_fullsized":"2024-04-25T11:20:25.702747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848256+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:25.848256+0000","last_peered":"2024-04-25T11:20:25.848256+0000","last_clean":"2024-04-25T11:20:25.848256+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:25.848256+0000","last_undegraded":"2024-04-25T11:20:25.848256+0000","last_fullsized":"2024-04-25T11:20:25.848256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849356+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:25.849356+0000","last_peered":"2024-04-25T11:20:25.849356+0000","last_clean":"2024-04-25T11:20:25.849356+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:25.849356+0000","last_undegraded":"2024-04-25T11:20:25.849356+0000","last_fullsized":"2024-04-25T11:20:25.849356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067076+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:25.067076+0000","last_peered":"2024-04-25T11:20:25.067076+0000","last_clean":"2024-04-25T11:20:25.067076+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:25.067076+0000","last_undegraded":"2024-04-25T11:20:25.067076+0000","last_fullsized":"2024-04-25T11:20:25.067076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851339+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:20:25.851339+0000","last_peered":"2024-04-25T11:20:25.851339+0000","last_clean":"2024-04-25T11:20:25.851339+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:20:25.851339+0000","last_undegraded":"2024-04-25T11:20:25.851339+0000","last_fullsized":"2024-04-25T11:20:25.851339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065892+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:25.065892+0000","last_peered":"2024-04-25T11:20:25.065892+0000","last_clean":"2024-04-25T11:20:25.065892+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:25.065892+0000","last_undegraded":"2024-04-25T11:20:25.065892+0000","last_fullsized":"2024-04-25T11:20:25.065892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852736+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:20:25.852736+0000","last_peered":"2024-04-25T11:20:25.852736+0000","last_clean":"2024-04-25T11:20:25.852736+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:20:25.852736+0000","last_undegraded":"2024-04-25T11:20:25.852736+0000","last_fullsized":"2024-04-25T11:20:25.852736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066385+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:25.066385+0000","last_peered":"2024-04-25T11:20:25.066385+0000","last_clean":"2024-04-25T11:20:25.066385+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:25.066385+0000","last_undegraded":"2024-04-25T11:20:25.066385+0000","last_fullsized":"2024-04-25T11:20:25.066385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849276+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:20:25.849276+0000","last_peered":"2024-04-25T11:20:25.849276+0000","last_clean":"2024-04-25T11:20:25.849276+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849276+0000","last_undegraded":"2024-04-25T11:20:25.849276+0000","last_fullsized":"2024-04-25T11:20:25.849276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703124+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:20:25.703124+0000","last_peered":"2024-04-25T11:20:25.703124+0000","last_clean":"2024-04-25T11:20:25.703124+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:20:25.703124+0000","last_undegraded":"2024-04-25T11:20:25.703124+0000","last_fullsized":"2024-04-25T11:20:25.703124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849275+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:20:25.849275+0000","last_peered":"2024-04-25T11:20:25.849275+0000","last_clean":"2024-04-25T11:20:25.849275+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:20:25.849275+0000","last_undegraded":"2024-04-25T11:20:25.849275+0000","last_fullsized":"2024-04-25T11:20:25.849275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703709+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:20:25.703709+0000","last_peered":"2024-04-25T11:20:25.703709+0000","last_clean":"2024-04-25T11:20:25.703709+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:20:25.703709+0000","last_undegraded":"2024-04-25T11:20:25.703709+0000","last_fullsized":"2024-04-25T11:20:25.703709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849944+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:25.849944+0000","last_peered":"2024-04-25T11:20:25.849944+0000","last_clean":"2024-04-25T11:20:25.849944+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:25.849944+0000","last_undegraded":"2024-04-25T11:20:25.849944+0000","last_fullsized":"2024-04-25T11:20:25.849944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852390+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:20:25.852390+0000","last_peered":"2024-04-25T11:20:25.852390+0000","last_clean":"2024-04-25T11:20:25.852390+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:20:25.852390+0000","last_undegraded":"2024-04-25T11:20:25.852390+0000","last_fullsized":"2024-04-25T11:20:25.852390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:20:58.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:20:58.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.740+0000 7f8d36e19640 1 Processor -- start 2024-04-25T11:20:58.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.740+0000 7f8d36e19640 1 -- start start 2024-04-25T11:20:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d36e19640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8d300c4750 0x7f8d30152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d36e19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d36e19640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d30071300 con 0x7f8d300736a0 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d36e19640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f8d300710d0 con 0x7f8d300c4750 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d36e19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d300c95f0 con 0x7f8d30158590 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d34bb5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8d300c4750 0x7f8d30152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d353b6640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8d300736a0 0x7f8d3015bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49762/0 (socket says 172.21.15.136:49762) 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d353b6640 1 -- 172.21.15.136:0/827228793 learned_addr learned my addr 172.21.15.136:0/827228793 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d2ffff640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d34bb5640 1 -- 172.21.15.136:0/827228793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 msgr2=0x7f8d301589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d34bb5640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d34bb5640 1 -- 172.21.15.136:0/827228793 >> v1:172.21.15.136:6789/0 conn(0x7f8d300736a0 legacy=0x7f8d3015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.744+0000 7f8d34bb5640 1 -- 172.21.15.136:0/827228793 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d300c7300 con 0x7f8d300c4750 2024-04-25T11:20:58.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d34bb5640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30152900 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097c0 tx=0x7f8d2001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44308e258ec793e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d2f7fe640 1 -- 172.21.15.136:0/827228793 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200b0070 con 0x7f8d300c4750 2024-04-25T11:20:58.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d2f7fe640 1 -- 172.21.15.136:0/827228793 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d200ab600 con 0x7f8d300c4750 2024-04-25T11:20:58.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d2f7fe640 1 -- 172.21.15.136:0/827228793 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200ab9d0 con 0x7f8d300c4750 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- 172.21.15.136:0/827228793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 msgr2=0x7f8d30152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30152900 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d200097c0 tx=0x7f8d2001c380 comp rx=0 tx=0).stop 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- 172.21.15.136:0/827228793 shutdown_connections 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 --2- 172.21.15.136:0/827228793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30152900 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- 172.21.15.136:0/827228793 >> 172.21.15.136:0/827228793 conn(0x7f8d300c9d50 msgr2=0x7f8d300ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- 172.21.15.136:0/827228793 shutdown_connections 2024-04-25T11:20:58.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- 172.21.15.136:0/827228793 wait complete. 2024-04-25T11:20:58.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 Processor -- start 2024-04-25T11:20:58.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.748+0000 7f8d36e19640 1 -- start start 2024-04-25T11:20:58.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d300736a0 0x7f8d301510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d300ca870 con 0x7f8d300c4750 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d300c9a80 con 0x7f8d300736a0 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d300ca9f0 con 0x7f8d30158590 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35276/0 (socket says 172.21.15.136:35276) 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 -- 172.21.15.136:0/327626126 learned_addr learned my addr 172.21.15.136:0/327626126 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d353b6640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d34bb5640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d300736a0 0x7f8d301510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 -- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 msgr2=0x7f8d301f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301f5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 -- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d300736a0 msgr2=0x7f8d301510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d300736a0 0x7f8d301510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 -- 172.21.15.136:0/327626126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d301f21a0 con 0x7f8d300c4750 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d353b6640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2ffff640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d24007e40 tx=0x7f8d2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24014620 con 0x7f8d300c4750 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d24019070 con 0x7f8d300c4750 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d301445e0 con 0x7f8d300c4750 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24014a00 con 0x7f8d300c4750 2024-04-25T11:20:58.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d301f2740 con 0x7f8d300c4750 2024-04-25T11:20:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d3006a350 con 0x7f8d300c4750 2024-04-25T11:20:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8d24014ba0 con 0x7f8d300c4750 2024-04-25T11:20:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.752+0000 7f8d2d7fa640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 0x7f8d1c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.756+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8d24090cb0 con 0x7f8d300c4750 2024-04-25T11:20:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.756+0000 7f8d34bb5640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 0x7f8d1c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:58.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.756+0000 7f8d34bb5640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 0x7f8d1c071910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8d200ad000 tx=0x7f8d20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:58.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.756+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8d2405d380 con 0x7f8d300c4750 2024-04-25T11:20:58.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.904+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d30144390 con 0x7f8d1c06f400 2024-04-25T11:20:58.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.948+0000 7f8d2d7fa640 1 -- 172.21.15.136:0/327626126 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332524 (secure 0 0 0) 0x7f8d30144390 con 0x7f8d1c06f400 2024-04-25T11:20:58.951 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:58.953 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:20:58.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 msgr2=0x7f8d1c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 0x7f8d1c071910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8d200ad000 tx=0x7f8d20008040 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 msgr2=0x7f8d30151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8d24007e40 tx=0x7f8d2400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 shutdown_connections 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8d1c06f400 0x7f8d1c071910 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d30158590 0x7f8d301f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d300c4750 0x7f8d30151610 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 --2- 172.21.15.136:0/327626126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d300736a0 0x7f8d301510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 >> 172.21.15.136:0/327626126 conn(0x7f8d300c9d50 msgr2=0x7f8d30157510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 shutdown_connections 2024-04-25T11:20:58.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:58.956+0000 7f8d36e19640 1 -- 172.21.15.136:0/327626126 wait complete. 2024-04-25T11:20:58.971 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-25T11:20:58.768235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":150,"num_write_kb":1647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236000,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738720,"statfs":{"total":773094113280,"available":772852449280,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005788"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070611+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:25.070611+0000","last_peered":"2024-04-25T11:20:25.070611+0000","last_clean":"2024-04-25T11:20:25.070611+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:25.070611+0000","last_undegraded":"2024-04-25T11:20:25.070611+0000","last_fullsized":"2024-04-25T11:20:25.070611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066703+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:25.066703+0000","last_peered":"2024-04-25T11:20:25.066703+0000","last_clean":"2024-04-25T11:20:25.066703+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:25.066703+0000","last_undegraded":"2024-04-25T11:20:25.066703+0000","last_fullsized":"2024-04-25T11:20:25.066703+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070831+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:25.070831+0000","last_peered":"2024-04-25T11:20:25.070831+0000","last_clean":"2024-04-25T11:20:25.070831+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:25.070831+0000","last_undegraded":"2024-04-25T11:20:25.070831+0000","last_fullsized":"2024-04-25T11:20:25.070831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848527+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:25.848527+0000","last_peered":"2024-04-25T11:20:25.848527+0000","last_clean":"2024-04-25T11:20:25.848527+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:25.848527+0000","last_undegraded":"2024-04-25T11:20:25.848527+0000","last_fullsized":"2024-04-25T11:20:25.848527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850005+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:20:25.850005+0000","last_peered":"2024-04-25T11:20:25.850005+0000","last_clean":"2024-04-25T11:20:25.850005+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:20:25.850005+0000","last_undegraded":"2024-04-25T11:20:25.850005+0000","last_fullsized":"2024-04-25T11:20:25.850005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852853+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:20:25.852853+0000","last_peered":"2024-04-25T11:20:25.852853+0000","last_clean":"2024-04-25T11:20:25.852853+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.852853+0000","last_undegraded":"2024-04-25T11:20:25.852853+0000","last_fullsized":"2024-04-25T11:20:25.852853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703772+0000","last_change":"2024-04-25T11:20:20.032982+0000","last_active":"2024-04-25T11:20:25.703772+0000","last_peered":"2024-04-25T11:20:25.703772+0000","last_clean":"2024-04-25T11:20:25.703772+0000","last_became_active":"2024-04-25T11:20:20.032127+0000","last_became_peered":"2024-04-25T11:20:20.032127+0000","last_unstale":"2024-04-25T11:20:25.703772+0000","last_undegraded":"2024-04-25T11:20:25.703772+0000","last_fullsized":"2024-04-25T11:20:25.703772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852476+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:20:25.852476+0000","last_peered":"2024-04-25T11:20:25.852476+0000","last_clean":"2024-04-25T11:20:25.852476+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:20:25.852476+0000","last_undegraded":"2024-04-25T11:20:25.852476+0000","last_fullsized":"2024-04-25T11:20:25.852476+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851378+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:20:25.851378+0000","last_peered":"2024-04-25T11:20:25.851378+0000","last_clean":"2024-04-25T11:20:25.851378+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:20:25.851378+0000","last_undegraded":"2024-04-25T11:20:25.851378+0000","last_fullsized":"2024-04-25T11:20:25.851378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704317+0000","last_change":"2024-04-25T11:20:20.031174+0000","last_active":"2024-04-25T11:20:25.704317+0000","last_peered":"2024-04-25T11:20:25.704317+0000","last_clean":"2024-04-25T11:20:25.704317+0000","last_became_active":"2024-04-25T11:20:20.030340+0000","last_became_peered":"2024-04-25T11:20:20.030340+0000","last_unstale":"2024-04-25T11:20:25.704317+0000","last_undegraded":"2024-04-25T11:20:25.704317+0000","last_fullsized":"2024-04-25T11:20:25.704317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850069+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:20:25.850069+0000","last_peered":"2024-04-25T11:20:25.850069+0000","last_clean":"2024-04-25T11:20:25.850069+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:20:25.850069+0000","last_undegraded":"2024-04-25T11:20:25.850069+0000","last_fullsized":"2024-04-25T11:20:25.850069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065389+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:25.065389+0000","last_peered":"2024-04-25T11:20:25.065389+0000","last_clean":"2024-04-25T11:20:25.065389+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:25.065389+0000","last_undegraded":"2024-04-25T11:20:25.065389+0000","last_fullsized":"2024-04-25T11:20:25.065389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850811+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:20:25.850811+0000","last_peered":"2024-04-25T11:20:25.850811+0000","last_clean":"2024-04-25T11:20:25.850811+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:20:25.850811+0000","last_undegraded":"2024-04-25T11:20:25.850811+0000","last_fullsized":"2024-04-25T11:20:25.850811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851666+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:20:25.851666+0000","last_peered":"2024-04-25T11:20:25.851666+0000","last_clean":"2024-04-25T11:20:25.851666+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:20:25.851666+0000","last_undegraded":"2024-04-25T11:20:25.851666+0000","last_fullsized":"2024-04-25T11:20:25.851666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067265+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:25.067265+0000","last_peered":"2024-04-25T11:20:25.067265+0000","last_clean":"2024-04-25T11:20:25.067265+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:25.067265+0000","last_undegraded":"2024-04-25T11:20:25.067265+0000","last_fullsized":"2024-04-25T11:20:25.067265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850338+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:20:25.850338+0000","last_peered":"2024-04-25T11:20:25.850338+0000","last_clean":"2024-04-25T11:20:25.850338+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:20:25.850338+0000","last_undegraded":"2024-04-25T11:20:25.850338+0000","last_fullsized":"2024-04-25T11:20:25.850338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850575+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:20:25.850575+0000","last_peered":"2024-04-25T11:20:25.850575+0000","last_clean":"2024-04-25T11:20:25.850575+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:20:25.850575+0000","last_undegraded":"2024-04-25T11:20:25.850575+0000","last_fullsized":"2024-04-25T11:20:25.850575+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066984+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:25.066984+0000","last_peered":"2024-04-25T11:20:25.066984+0000","last_clean":"2024-04-25T11:20:25.066984+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:25.066984+0000","last_undegraded":"2024-04-25T11:20:25.066984+0000","last_fullsized":"2024-04-25T11:20:25.066984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848318+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:20:25.848318+0000","last_peered":"2024-04-25T11:20:25.848318+0000","last_clean":"2024-04-25T11:20:25.848318+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:20:25.848318+0000","last_undegraded":"2024-04-25T11:20:25.848318+0000","last_fullsized":"2024-04-25T11:20:25.848318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850089+0000","last_change":"2024-04-25T11:20:24.165156+0000","last_active":"2024-04-25T11:20:25.850089+0000","last_peered":"2024-04-25T11:20:25.850089+0000","last_clean":"2024-04-25T11:20:25.850089+0000","last_became_active":"2024-04-25T11:20:24.164587+0000","last_became_peered":"2024-04-25T11:20:24.164587+0000","last_unstale":"2024-04-25T11:20:25.850089+0000","last_undegraded":"2024-04-25T11:20:25.850089+0000","last_fullsized":"2024-04-25T11:20:25.850089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851436+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:20:25.851436+0000","last_peered":"2024-04-25T11:20:25.851436+0000","last_clean":"2024-04-25T11:20:25.851436+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:20:25.851436+0000","last_undegraded":"2024-04-25T11:20:25.851436+0000","last_fullsized":"2024-04-25T11:20:25.851436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849318+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:20:25.849318+0000","last_peered":"2024-04-25T11:20:25.849318+0000","last_clean":"2024-04-25T11:20:25.849318+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:20:25.849318+0000","last_undegraded":"2024-04-25T11:20:25.849318+0000","last_fullsized":"2024-04-25T11:20:25.849318+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850973+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:20:25.850973+0000","last_peered":"2024-04-25T11:20:25.850973+0000","last_clean":"2024-04-25T11:20:25.850973+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:20:25.850973+0000","last_undegraded":"2024-04-25T11:20:25.850973+0000","last_fullsized":"2024-04-25T11:20:25.850973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849979+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:25.849979+0000","last_peered":"2024-04-25T11:20:25.849979+0000","last_clean":"2024-04-25T11:20:25.849979+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:25.849979+0000","last_undegraded":"2024-04-25T11:20:25.849979+0000","last_fullsized":"2024-04-25T11:20:25.849979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066481+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:20:25.066481+0000","last_peered":"2024-04-25T11:20:25.066481+0000","last_clean":"2024-04-25T11:20:25.066481+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:20:25.066481+0000","last_undegraded":"2024-04-25T11:20:25.066481+0000","last_fullsized":"2024-04-25T11:20:25.066481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850564+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:20:25.850564+0000","last_peered":"2024-04-25T11:20:25.850564+0000","last_clean":"2024-04-25T11:20:25.850564+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:20:25.850564+0000","last_undegraded":"2024-04-25T11:20:25.850564+0000","last_fullsized":"2024-04-25T11:20:25.850564+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071084+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:25.071084+0000","last_peered":"2024-04-25T11:20:25.071084+0000","last_clean":"2024-04-25T11:20:25.071084+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:25.071084+0000","last_undegraded":"2024-04-25T11:20:25.071084+0000","last_fullsized":"2024-04-25T11:20:25.071084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850091+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:20:25.850091+0000","last_peered":"2024-04-25T11:20:25.850091+0000","last_clean":"2024-04-25T11:20:25.850091+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:20:25.850091+0000","last_undegraded":"2024-04-25T11:20:25.850091+0000","last_fullsized":"2024-04-25T11:20:25.850091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066330+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:25.066330+0000","last_peered":"2024-04-25T11:20:25.066330+0000","last_clean":"2024-04-25T11:20:25.066330+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:25.066330+0000","last_undegraded":"2024-04-25T11:20:25.066330+0000","last_fullsized":"2024-04-25T11:20:25.066330+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850885+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:25.850885+0000","last_peered":"2024-04-25T11:20:25.850885+0000","last_clean":"2024-04-25T11:20:25.850885+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:25.850885+0000","last_undegraded":"2024-04-25T11:20:25.850885+0000","last_fullsized":"2024-04-25T11:20:25.850885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853257+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:20:25.853257+0000","last_peered":"2024-04-25T11:20:25.853257+0000","last_clean":"2024-04-25T11:20:25.853257+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:20:25.853257+0000","last_undegraded":"2024-04-25T11:20:25.853257+0000","last_fullsized":"2024-04-25T11:20:25.853257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848798+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:25.848798+0000","last_peered":"2024-04-25T11:20:25.848798+0000","last_clean":"2024-04-25T11:20:25.848798+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:25.848798+0000","last_undegraded":"2024-04-25T11:20:25.848798+0000","last_fullsized":"2024-04-25T11:20:25.848798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704541+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.704541+0000","last_peered":"2024-04-25T11:20:25.704541+0000","last_clean":"2024-04-25T11:20:25.704541+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:20:25.704541+0000","last_undegraded":"2024-04-25T11:20:25.704541+0000","last_fullsized":"2024-04-25T11:20:25.704541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850957+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:25.850957+0000","last_peered":"2024-04-25T11:20:25.850957+0000","last_clean":"2024-04-25T11:20:25.850957+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:25.850957+0000","last_undegraded":"2024-04-25T11:20:25.850957+0000","last_fullsized":"2024-04-25T11:20:25.850957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848973+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:25.848973+0000","last_peered":"2024-04-25T11:20:25.848973+0000","last_clean":"2024-04-25T11:20:25.848973+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:25.848973+0000","last_undegraded":"2024-04-25T11:20:25.848973+0000","last_fullsized":"2024-04-25T11:20:25.848973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852994+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:20:25.852994+0000","last_peered":"2024-04-25T11:20:25.852994+0000","last_clean":"2024-04-25T11:20:25.852994+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:20:25.852994+0000","last_undegraded":"2024-04-25T11:20:25.852994+0000","last_fullsized":"2024-04-25T11:20:25.852994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703782+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:20:25.703782+0000","last_peered":"2024-04-25T11:20:25.703782+0000","last_clean":"2024-04-25T11:20:25.703782+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:20:25.703782+0000","last_undegraded":"2024-04-25T11:20:25.703782+0000","last_fullsized":"2024-04-25T11:20:25.703782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850807+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:25.850807+0000","last_peered":"2024-04-25T11:20:25.850807+0000","last_clean":"2024-04-25T11:20:25.850807+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:25.850807+0000","last_undegraded":"2024-04-25T11:20:25.850807+0000","last_fullsized":"2024-04-25T11:20:25.850807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850780+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:25.850780+0000","last_peered":"2024-04-25T11:20:25.850780+0000","last_clean":"2024-04-25T11:20:25.850780+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:25.850780+0000","last_undegraded":"2024-04-25T11:20:25.850780+0000","last_fullsized":"2024-04-25T11:20:25.850780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066095+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:20:25.066095+0000","last_peered":"2024-04-25T11:20:25.066095+0000","last_clean":"2024-04-25T11:20:25.066095+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:20:25.066095+0000","last_undegraded":"2024-04-25T11:20:25.066095+0000","last_fullsized":"2024-04-25T11:20:25.066095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850239+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:25.850239+0000","last_peered":"2024-04-25T11:20:25.850239+0000","last_clean":"2024-04-25T11:20:25.850239+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:25.850239+0000","last_undegraded":"2024-04-25T11:20:25.850239+0000","last_fullsized":"2024-04-25T11:20:25.850239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848175+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:25.848175+0000","last_peered":"2024-04-25T11:20:25.848175+0000","last_clean":"2024-04-25T11:20:25.848175+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:25.848175+0000","last_undegraded":"2024-04-25T11:20:25.848175+0000","last_fullsized":"2024-04-25T11:20:25.848175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849105+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:20:25.849105+0000","last_peered":"2024-04-25T11:20:25.849105+0000","last_clean":"2024-04-25T11:20:25.849105+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:20:25.849105+0000","last_undegraded":"2024-04-25T11:20:25.849105+0000","last_fullsized":"2024-04-25T11:20:25.849105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849839+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:25.849839+0000","last_peered":"2024-04-25T11:20:25.849839+0000","last_clean":"2024-04-25T11:20:25.849839+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:25.849839+0000","last_undegraded":"2024-04-25T11:20:25.849839+0000","last_fullsized":"2024-04-25T11:20:25.849839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066978+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:25.066978+0000","last_peered":"2024-04-25T11:20:25.066978+0000","last_clean":"2024-04-25T11:20:25.066978+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:25.066978+0000","last_undegraded":"2024-04-25T11:20:25.066978+0000","last_fullsized":"2024-04-25T11:20:25.066978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703462+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:20:25.703462+0000","last_peered":"2024-04-25T11:20:25.703462+0000","last_clean":"2024-04-25T11:20:25.703462+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:20:25.703462+0000","last_undegraded":"2024-04-25T11:20:25.703462+0000","last_fullsized":"2024-04-25T11:20:25.703462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849435+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:20:25.849435+0000","last_peered":"2024-04-25T11:20:25.849435+0000","last_clean":"2024-04-25T11:20:25.849435+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:20:25.849435+0000","last_undegraded":"2024-04-25T11:20:25.849435+0000","last_fullsized":"2024-04-25T11:20:25.849435+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066131+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:25.066131+0000","last_peered":"2024-04-25T11:20:25.066131+0000","last_clean":"2024-04-25T11:20:25.066131+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:25.066131+0000","last_undegraded":"2024-04-25T11:20:25.066131+0000","last_fullsized":"2024-04-25T11:20:25.066131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852310+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:20:25.852310+0000","last_peered":"2024-04-25T11:20:25.852310+0000","last_clean":"2024-04-25T11:20:25.852310+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:20:25.852310+0000","last_undegraded":"2024-04-25T11:20:25.852310+0000","last_fullsized":"2024-04-25T11:20:25.852310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065502+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:25.065502+0000","last_peered":"2024-04-25T11:20:25.065502+0000","last_clean":"2024-04-25T11:20:25.065502+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:25.065502+0000","last_undegraded":"2024-04-25T11:20:25.065502+0000","last_fullsized":"2024-04-25T11:20:25.065502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703910+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:20:25.703910+0000","last_peered":"2024-04-25T11:20:25.703910+0000","last_clean":"2024-04-25T11:20:25.703910+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:20:25.703910+0000","last_undegraded":"2024-04-25T11:20:25.703910+0000","last_fullsized":"2024-04-25T11:20:25.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849639+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:20:25.849639+0000","last_peered":"2024-04-25T11:20:25.849639+0000","last_clean":"2024-04-25T11:20:25.849639+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:20:25.849639+0000","last_undegraded":"2024-04-25T11:20:25.849639+0000","last_fullsized":"2024-04-25T11:20:25.849639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849150+0000","last_change":"2024-04-25T11:20:24.098651+0000","last_active":"2024-04-25T11:20:25.849150+0000","last_peered":"2024-04-25T11:20:25.849150+0000","last_clean":"2024-04-25T11:20:25.849150+0000","last_became_active":"2024-04-25T11:20:24.097985+0000","last_became_peered":"2024-04-25T11:20:24.097985+0000","last_unstale":"2024-04-25T11:20:25.849150+0000","last_undegraded":"2024-04-25T11:20:25.849150+0000","last_fullsized":"2024-04-25T11:20:25.849150+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:39.606818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852724+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:20:25.852724+0000","last_peered":"2024-04-25T11:20:25.852724+0000","last_clean":"2024-04-25T11:20:25.852724+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:20:25.852724+0000","last_undegraded":"2024-04-25T11:20:25.852724+0000","last_fullsized":"2024-04-25T11:20:25.852724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067792+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:20:25.067792+0000","last_peered":"2024-04-25T11:20:25.067792+0000","last_clean":"2024-04-25T11:20:25.067792+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:20:25.067792+0000","last_undegraded":"2024-04-25T11:20:25.067792+0000","last_fullsized":"2024-04-25T11:20:25.067792+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702944+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:20:25.702944+0000","last_peered":"2024-04-25T11:20:25.702944+0000","last_clean":"2024-04-25T11:20:25.702944+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:20:25.702944+0000","last_undegraded":"2024-04-25T11:20:25.702944+0000","last_fullsized":"2024-04-25T11:20:25.702944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852444+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:20:25.852444+0000","last_peered":"2024-04-25T11:20:25.852444+0000","last_clean":"2024-04-25T11:20:25.852444+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:20:25.852444+0000","last_undegraded":"2024-04-25T11:20:25.852444+0000","last_fullsized":"2024-04-25T11:20:25.852444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851726+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:20:25.851726+0000","last_peered":"2024-04-25T11:20:25.851726+0000","last_clean":"2024-04-25T11:20:25.851726+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:20:25.851726+0000","last_undegraded":"2024-04-25T11:20:25.851726+0000","last_fullsized":"2024-04-25T11:20:25.851726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850209+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:20:25.850209+0000","last_peered":"2024-04-25T11:20:25.850209+0000","last_clean":"2024-04-25T11:20:25.850209+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:20:25.850209+0000","last_undegraded":"2024-04-25T11:20:25.850209+0000","last_fullsized":"2024-04-25T11:20:25.850209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849636+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:20:25.849636+0000","last_peered":"2024-04-25T11:20:25.849636+0000","last_clean":"2024-04-25T11:20:25.849636+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:20:25.849636+0000","last_undegraded":"2024-04-25T11:20:25.849636+0000","last_fullsized":"2024-04-25T11:20:25.849636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065250+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:20:25.065250+0000","last_peered":"2024-04-25T11:20:25.065250+0000","last_clean":"2024-04-25T11:20:25.065250+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:20:25.065250+0000","last_undegraded":"2024-04-25T11:20:25.065250+0000","last_fullsized":"2024-04-25T11:20:25.065250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849363+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:25.849363+0000","last_peered":"2024-04-25T11:20:25.849363+0000","last_clean":"2024-04-25T11:20:25.849363+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:25.849363+0000","last_undegraded":"2024-04-25T11:20:25.849363+0000","last_fullsized":"2024-04-25T11:20:25.849363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065441+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:25.065441+0000","last_peered":"2024-04-25T11:20:25.065441+0000","last_clean":"2024-04-25T11:20:25.065441+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:25.065441+0000","last_undegraded":"2024-04-25T11:20:25.065441+0000","last_fullsized":"2024-04-25T11:20:25.065441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065600+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:20:25.065600+0000","last_peered":"2024-04-25T11:20:25.065600+0000","last_clean":"2024-04-25T11:20:25.065600+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:20:25.065600+0000","last_undegraded":"2024-04-25T11:20:25.065600+0000","last_fullsized":"2024-04-25T11:20:25.065600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.071194+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:25.071194+0000","last_peered":"2024-04-25T11:20:25.071194+0000","last_clean":"2024-04-25T11:20:25.071194+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:25.071194+0000","last_undegraded":"2024-04-25T11:20:25.071194+0000","last_fullsized":"2024-04-25T11:20:25.071194+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850312+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:20:25.850312+0000","last_peered":"2024-04-25T11:20:25.850312+0000","last_clean":"2024-04-25T11:20:25.850312+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:20:25.850312+0000","last_undegraded":"2024-04-25T11:20:25.850312+0000","last_fullsized":"2024-04-25T11:20:25.850312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703560+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:20:25.703560+0000","last_peered":"2024-04-25T11:20:25.703560+0000","last_clean":"2024-04-25T11:20:25.703560+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:20:25.703560+0000","last_undegraded":"2024-04-25T11:20:25.703560+0000","last_fullsized":"2024-04-25T11:20:25.703560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067907+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:25.067907+0000","last_peered":"2024-04-25T11:20:25.067907+0000","last_clean":"2024-04-25T11:20:25.067907+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:25.067907+0000","last_undegraded":"2024-04-25T11:20:25.067907+0000","last_fullsized":"2024-04-25T11:20:25.067907+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849479+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:25.849479+0000","last_peered":"2024-04-25T11:20:25.849479+0000","last_clean":"2024-04-25T11:20:25.849479+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:25.849479+0000","last_undegraded":"2024-04-25T11:20:25.849479+0000","last_fullsized":"2024-04-25T11:20:25.849479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849577+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:20:25.849577+0000","last_peered":"2024-04-25T11:20:25.849577+0000","last_clean":"2024-04-25T11:20:25.849577+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:20:25.849577+0000","last_undegraded":"2024-04-25T11:20:25.849577+0000","last_fullsized":"2024-04-25T11:20:25.849577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704488+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:20:25.704488+0000","last_peered":"2024-04-25T11:20:25.704488+0000","last_clean":"2024-04-25T11:20:25.704488+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:20:25.704488+0000","last_undegraded":"2024-04-25T11:20:25.704488+0000","last_fullsized":"2024-04-25T11:20:25.704488+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851990+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:20:25.851990+0000","last_peered":"2024-04-25T11:20:25.851990+0000","last_clean":"2024-04-25T11:20:25.851990+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:20:25.851990+0000","last_undegraded":"2024-04-25T11:20:25.851990+0000","last_fullsized":"2024-04-25T11:20:25.851990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:41.996974+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:20:41.996974+0000","last_peered":"2024-04-25T11:20:41.996974+0000","last_clean":"2024-04-25T11:20:41.996974+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:20:41.996974+0000","last_undegraded":"2024-04-25T11:20:41.996974+0000","last_fullsized":"2024-04-25T11:20:41.996974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849162+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:25.849162+0000","last_peered":"2024-04-25T11:20:25.849162+0000","last_clean":"2024-04-25T11:20:25.849162+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:25.849162+0000","last_undegraded":"2024-04-25T11:20:25.849162+0000","last_fullsized":"2024-04-25T11:20:25.849162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851639+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:20:25.851639+0000","last_peered":"2024-04-25T11:20:25.851639+0000","last_clean":"2024-04-25T11:20:25.851639+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:20:25.851639+0000","last_undegraded":"2024-04-25T11:20:25.851639+0000","last_fullsized":"2024-04-25T11:20:25.851639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849514+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:25.849514+0000","last_peered":"2024-04-25T11:20:25.849514+0000","last_clean":"2024-04-25T11:20:25.849514+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:25.849514+0000","last_undegraded":"2024-04-25T11:20:25.849514+0000","last_fullsized":"2024-04-25T11:20:25.849514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852817+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:20:25.852817+0000","last_peered":"2024-04-25T11:20:25.852817+0000","last_clean":"2024-04-25T11:20:25.852817+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:20:25.852817+0000","last_undegraded":"2024-04-25T11:20:25.852817+0000","last_fullsized":"2024-04-25T11:20:25.852817+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848647+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:25.848647+0000","last_peered":"2024-04-25T11:20:25.848647+0000","last_clean":"2024-04-25T11:20:25.848647+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:25.848647+0000","last_undegraded":"2024-04-25T11:20:25.848647+0000","last_fullsized":"2024-04-25T11:20:25.848647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704293+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:20:25.704293+0000","last_peered":"2024-04-25T11:20:25.704293+0000","last_clean":"2024-04-25T11:20:25.704293+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:20:25.704293+0000","last_undegraded":"2024-04-25T11:20:25.704293+0000","last_fullsized":"2024-04-25T11:20:25.704293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066858+0000","last_change":"2024-04-25T11:20:24.093677+0000","last_active":"2024-04-25T11:20:25.066858+0000","last_peered":"2024-04-25T11:20:25.066858+0000","last_clean":"2024-04-25T11:20:25.066858+0000","last_became_active":"2024-04-25T11:20:24.092907+0000","last_became_peered":"2024-04-25T11:20:24.092907+0000","last_unstale":"2024-04-25T11:20:25.066858+0000","last_undegraded":"2024-04-25T11:20:25.066858+0000","last_fullsized":"2024-04-25T11:20:25.066858+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:30.768495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704129+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:20:25.704129+0000","last_peered":"2024-04-25T11:20:25.704129+0000","last_clean":"2024-04-25T11:20:25.704129+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:20:25.704129+0000","last_undegraded":"2024-04-25T11:20:25.704129+0000","last_fullsized":"2024-04-25T11:20:25.704129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066326+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:20:25.066326+0000","last_peered":"2024-04-25T11:20:25.066326+0000","last_clean":"2024-04-25T11:20:25.066326+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:20:25.066326+0000","last_undegraded":"2024-04-25T11:20:25.066326+0000","last_fullsized":"2024-04-25T11:20:25.066326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067316+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:25.067316+0000","last_peered":"2024-04-25T11:20:25.067316+0000","last_clean":"2024-04-25T11:20:25.067316+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:25.067316+0000","last_undegraded":"2024-04-25T11:20:25.067316+0000","last_fullsized":"2024-04-25T11:20:25.067316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848888+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:25.848888+0000","last_peered":"2024-04-25T11:20:25.848888+0000","last_clean":"2024-04-25T11:20:25.848888+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:25.848888+0000","last_undegraded":"2024-04-25T11:20:25.848888+0000","last_fullsized":"2024-04-25T11:20:25.848888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848177+0000","last_change":"2024-04-25T11:20:20.040078+0000","last_active":"2024-04-25T11:20:25.848177+0000","last_peered":"2024-04-25T11:20:25.848177+0000","last_clean":"2024-04-25T11:20:25.848177+0000","last_became_active":"2024-04-25T11:20:20.039255+0000","last_became_peered":"2024-04-25T11:20:20.039255+0000","last_unstale":"2024-04-25T11:20:25.848177+0000","last_undegraded":"2024-04-25T11:20:25.848177+0000","last_fullsized":"2024-04-25T11:20:25.848177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703263+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:20:25.703263+0000","last_peered":"2024-04-25T11:20:25.703263+0000","last_clean":"2024-04-25T11:20:25.703263+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:20:25.703263+0000","last_undegraded":"2024-04-25T11:20:25.703263+0000","last_fullsized":"2024-04-25T11:20:25.703263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066138+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:25.066138+0000","last_peered":"2024-04-25T11:20:25.066138+0000","last_clean":"2024-04-25T11:20:25.066138+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:25.066138+0000","last_undegraded":"2024-04-25T11:20:25.066138+0000","last_fullsized":"2024-04-25T11:20:25.066138+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066710+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:25.066710+0000","last_peered":"2024-04-25T11:20:25.066710+0000","last_clean":"2024-04-25T11:20:25.066710+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:25.066710+0000","last_undegraded":"2024-04-25T11:20:25.066710+0000","last_fullsized":"2024-04-25T11:20:25.066710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067819+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:20:25.067819+0000","last_peered":"2024-04-25T11:20:25.067819+0000","last_clean":"2024-04-25T11:20:25.067819+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:20:25.067819+0000","last_undegraded":"2024-04-25T11:20:25.067819+0000","last_fullsized":"2024-04-25T11:20:25.067819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065961+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:20:25.065961+0000","last_peered":"2024-04-25T11:20:25.065961+0000","last_clean":"2024-04-25T11:20:25.065961+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:20:25.065961+0000","last_undegraded":"2024-04-25T11:20:25.065961+0000","last_fullsized":"2024-04-25T11:20:25.065961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849428+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:25.849428+0000","last_peered":"2024-04-25T11:20:25.849428+0000","last_clean":"2024-04-25T11:20:25.849428+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:25.849428+0000","last_undegraded":"2024-04-25T11:20:25.849428+0000","last_fullsized":"2024-04-25T11:20:25.849428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066149+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:25.066149+0000","last_peered":"2024-04-25T11:20:25.066149+0000","last_clean":"2024-04-25T11:20:25.066149+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:25.066149+0000","last_undegraded":"2024-04-25T11:20:25.066149+0000","last_fullsized":"2024-04-25T11:20:25.066149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067406+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:25.067406+0000","last_peered":"2024-04-25T11:20:25.067406+0000","last_clean":"2024-04-25T11:20:25.067406+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:25.067406+0000","last_undegraded":"2024-04-25T11:20:25.067406+0000","last_fullsized":"2024-04-25T11:20:25.067406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.850698+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:20:25.850698+0000","last_peered":"2024-04-25T11:20:25.850698+0000","last_clean":"2024-04-25T11:20:25.850698+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:20:25.850698+0000","last_undegraded":"2024-04-25T11:20:25.850698+0000","last_fullsized":"2024-04-25T11:20:25.850698+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070455+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:25.070455+0000","last_peered":"2024-04-25T11:20:25.070455+0000","last_clean":"2024-04-25T11:20:25.070455+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:25.070455+0000","last_undegraded":"2024-04-25T11:20:25.070455+0000","last_fullsized":"2024-04-25T11:20:25.070455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065240+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:25.065240+0000","last_peered":"2024-04-25T11:20:25.065240+0000","last_clean":"2024-04-25T11:20:25.065240+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:25.065240+0000","last_undegraded":"2024-04-25T11:20:25.065240+0000","last_fullsized":"2024-04-25T11:20:25.065240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065744+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:25.065744+0000","last_peered":"2024-04-25T11:20:25.065744+0000","last_clean":"2024-04-25T11:20:25.065744+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:25.065744+0000","last_undegraded":"2024-04-25T11:20:25.065744+0000","last_fullsized":"2024-04-25T11:20:25.065744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703495+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:20:25.703495+0000","last_peered":"2024-04-25T11:20:25.703495+0000","last_clean":"2024-04-25T11:20:25.703495+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:20:25.703495+0000","last_undegraded":"2024-04-25T11:20:25.703495+0000","last_fullsized":"2024-04-25T11:20:25.703495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849293+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:25.849293+0000","last_peered":"2024-04-25T11:20:25.849293+0000","last_clean":"2024-04-25T11:20:25.849293+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:25.849293+0000","last_undegraded":"2024-04-25T11:20:25.849293+0000","last_fullsized":"2024-04-25T11:20:25.849293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.068272+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:25.068272+0000","last_peered":"2024-04-25T11:20:25.068272+0000","last_clean":"2024-04-25T11:20:25.068272+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:25.068272+0000","last_undegraded":"2024-04-25T11:20:25.068272+0000","last_fullsized":"2024-04-25T11:20:25.068272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849313+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:25.849313+0000","last_peered":"2024-04-25T11:20:25.849313+0000","last_clean":"2024-04-25T11:20:25.849313+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849313+0000","last_undegraded":"2024-04-25T11:20:25.849313+0000","last_fullsized":"2024-04-25T11:20:25.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066579+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:25.066579+0000","last_peered":"2024-04-25T11:20:25.066579+0000","last_clean":"2024-04-25T11:20:25.066579+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:25.066579+0000","last_undegraded":"2024-04-25T11:20:25.066579+0000","last_fullsized":"2024-04-25T11:20:25.066579+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.070896+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:25.070896+0000","last_peered":"2024-04-25T11:20:25.070896+0000","last_clean":"2024-04-25T11:20:25.070896+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:25.070896+0000","last_undegraded":"2024-04-25T11:20:25.070896+0000","last_fullsized":"2024-04-25T11:20:25.070896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703122+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:20:25.703122+0000","last_peered":"2024-04-25T11:20:25.703122+0000","last_clean":"2024-04-25T11:20:25.703122+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:20:25.703122+0000","last_undegraded":"2024-04-25T11:20:25.703122+0000","last_fullsized":"2024-04-25T11:20:25.703122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067675+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:25.067675+0000","last_peered":"2024-04-25T11:20:25.067675+0000","last_clean":"2024-04-25T11:20:25.067675+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:25.067675+0000","last_undegraded":"2024-04-25T11:20:25.067675+0000","last_fullsized":"2024-04-25T11:20:25.067675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703259+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:20:25.703259+0000","last_peered":"2024-04-25T11:20:25.703259+0000","last_clean":"2024-04-25T11:20:25.703259+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:20:25.703259+0000","last_undegraded":"2024-04-25T11:20:25.703259+0000","last_fullsized":"2024-04-25T11:20:25.703259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848929+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:25.848929+0000","last_peered":"2024-04-25T11:20:25.848929+0000","last_clean":"2024-04-25T11:20:25.848929+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:25.848929+0000","last_undegraded":"2024-04-25T11:20:25.848929+0000","last_fullsized":"2024-04-25T11:20:25.848929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066246+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:25.066246+0000","last_peered":"2024-04-25T11:20:25.066246+0000","last_clean":"2024-04-25T11:20:25.066246+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:25.066246+0000","last_undegraded":"2024-04-25T11:20:25.066246+0000","last_fullsized":"2024-04-25T11:20:25.066246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.853167+0000","last_change":"2024-04-25T11:20:20.039980+0000","last_active":"2024-04-25T11:20:25.853167+0000","last_peered":"2024-04-25T11:20:25.853167+0000","last_clean":"2024-04-25T11:20:25.853167+0000","last_became_active":"2024-04-25T11:20:20.039297+0000","last_became_peered":"2024-04-25T11:20:20.039297+0000","last_unstale":"2024-04-25T11:20:25.853167+0000","last_undegraded":"2024-04-25T11:20:25.853167+0000","last_fullsized":"2024-04-25T11:20:25.853167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:18.780733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067502+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:25.067502+0000","last_peered":"2024-04-25T11:20:25.067502+0000","last_clean":"2024-04-25T11:20:25.067502+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:25.067502+0000","last_undegraded":"2024-04-25T11:20:25.067502+0000","last_fullsized":"2024-04-25T11:20:25.067502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.704854+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:20:25.704854+0000","last_peered":"2024-04-25T11:20:25.704854+0000","last_clean":"2024-04-25T11:20:25.704854+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:20:25.704854+0000","last_undegraded":"2024-04-25T11:20:25.704854+0000","last_fullsized":"2024-04-25T11:20:25.704854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.702747+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:20:25.702747+0000","last_peered":"2024-04-25T11:20:25.702747+0000","last_clean":"2024-04-25T11:20:25.702747+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:20:25.702747+0000","last_undegraded":"2024-04-25T11:20:25.702747+0000","last_fullsized":"2024-04-25T11:20:25.702747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848256+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:25.848256+0000","last_peered":"2024-04-25T11:20:25.848256+0000","last_clean":"2024-04-25T11:20:25.848256+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:25.848256+0000","last_undegraded":"2024-04-25T11:20:25.848256+0000","last_fullsized":"2024-04-25T11:20:25.848256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849356+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:25.849356+0000","last_peered":"2024-04-25T11:20:25.849356+0000","last_clean":"2024-04-25T11:20:25.849356+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:25.849356+0000","last_undegraded":"2024-04-25T11:20:25.849356+0000","last_fullsized":"2024-04-25T11:20:25.849356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.067076+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:25.067076+0000","last_peered":"2024-04-25T11:20:25.067076+0000","last_clean":"2024-04-25T11:20:25.067076+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:25.067076+0000","last_undegraded":"2024-04-25T11:20:25.067076+0000","last_fullsized":"2024-04-25T11:20:25.067076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.851339+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:20:25.851339+0000","last_peered":"2024-04-25T11:20:25.851339+0000","last_clean":"2024-04-25T11:20:25.851339+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:20:25.851339+0000","last_undegraded":"2024-04-25T11:20:25.851339+0000","last_fullsized":"2024-04-25T11:20:25.851339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.065892+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:25.065892+0000","last_peered":"2024-04-25T11:20:25.065892+0000","last_clean":"2024-04-25T11:20:25.065892+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:25.065892+0000","last_undegraded":"2024-04-25T11:20:25.065892+0000","last_fullsized":"2024-04-25T11:20:25.065892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066412+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:25.066412+0000","last_peered":"2024-04-25T11:20:25.066412+0000","last_clean":"2024-04-25T11:20:25.066412+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:25.066412+0000","last_undegraded":"2024-04-25T11:20:25.066412+0000","last_fullsized":"2024-04-25T11:20:25.066412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852736+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:20:25.852736+0000","last_peered":"2024-04-25T11:20:25.852736+0000","last_clean":"2024-04-25T11:20:25.852736+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:20:25.852736+0000","last_undegraded":"2024-04-25T11:20:25.852736+0000","last_fullsized":"2024-04-25T11:20:25.852736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.066385+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:25.066385+0000","last_peered":"2024-04-25T11:20:25.066385+0000","last_clean":"2024-04-25T11:20:25.066385+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:25.066385+0000","last_undegraded":"2024-04-25T11:20:25.066385+0000","last_fullsized":"2024-04-25T11:20:25.066385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849276+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:20:25.849276+0000","last_peered":"2024-04-25T11:20:25.849276+0000","last_clean":"2024-04-25T11:20:25.849276+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:25.849276+0000","last_undegraded":"2024-04-25T11:20:25.849276+0000","last_fullsized":"2024-04-25T11:20:25.849276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703124+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:20:25.703124+0000","last_peered":"2024-04-25T11:20:25.703124+0000","last_clean":"2024-04-25T11:20:25.703124+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:20:25.703124+0000","last_undegraded":"2024-04-25T11:20:25.703124+0000","last_fullsized":"2024-04-25T11:20:25.703124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849275+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:20:25.849275+0000","last_peered":"2024-04-25T11:20:25.849275+0000","last_clean":"2024-04-25T11:20:25.849275+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:20:25.849275+0000","last_undegraded":"2024-04-25T11:20:25.849275+0000","last_fullsized":"2024-04-25T11:20:25.849275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.703709+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:20:25.703709+0000","last_peered":"2024-04-25T11:20:25.703709+0000","last_clean":"2024-04-25T11:20:25.703709+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:20:25.703709+0000","last_undegraded":"2024-04-25T11:20:25.703709+0000","last_fullsized":"2024-04-25T11:20:25.703709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.848717+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:25.848717+0000","last_peered":"2024-04-25T11:20:25.848717+0000","last_clean":"2024-04-25T11:20:25.848717+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:25.848717+0000","last_undegraded":"2024-04-25T11:20:25.848717+0000","last_fullsized":"2024-04-25T11:20:25.848717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.849944+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:25.849944+0000","last_peered":"2024-04-25T11:20:25.849944+0000","last_clean":"2024-04-25T11:20:25.849944+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:25.849944+0000","last_undegraded":"2024-04-25T11:20:25.849944+0000","last_fullsized":"2024-04-25T11:20:25.849944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T11:20:25.852390+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:20:25.852390+0000","last_peered":"2024-04-25T11:20:25.852390+0000","last_clean":"2024-04-25T11:20:25.852390+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:20:25.852390+0000","last_undegraded":"2024-04-25T11:20:25.852390+0000","last_fullsized":"2024-04-25T11:20:25.852390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:20:58.976 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T11:20:58.976 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T11:20:58.976 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T11:20:58.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T11:20:59.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.152+0000 7fb178dcf640 1 Processor -- start 2024-04-25T11:20:59.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.152+0000 7fb178dcf640 1 -- start start 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb178dcf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 0x7fb17415d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb178dcf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb174070870 0x7fb1740c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb178dcf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb174143580 con 0x7fb174070c80 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb178dcf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb17415dd20 con 0x7fb174070870 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb178dcf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb17415e000 con 0x7fb174143170 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb172d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb174070c80 0x7fb174159e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49772/0 (socket says 172.21.15.136:49772) 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb172d9d640 1 -- 172.21.15.136:0/1971491608 learned_addr learned my addr 172.21.15.136:0/1971491608 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17259c640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 0x7fb17415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 --2- 172.21.15.136:0/1971491608 >> v2:172.21.15.157:3300/0 conn(0x7fb174070870 0x7fb1740c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1963934243 0 0) 0x7fb16c001120 con 0x7fb174070c80 2024-04-25T11:20:59.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb15c003530 con 0x7fb174070c80 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3069435069 0 0) 0x7fb15c003530 con 0x7fb174070c80 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 -- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 msgr2=0x7fb17415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 0x7fb17415d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 -- 172.21.15.136:0/1971491608 >> v1:172.21.15.136:6789/0 conn(0x7fb174070c80 legacy=0x7fb174159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 -- 172.21.15.136:0/1971491608 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17406fd90 con 0x7fb174070870 2024-04-25T11:20:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17259c640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 0x7fb17415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:20:59.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb171d9b640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070870 0x7fb1740c6960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097c0 tx=0x7fb16801ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec457f43dc39954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:59.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1680b5070 con 0x7fb174070870 2024-04-25T11:20:59.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1680a85f0 con 0x7fb174070870 2024-04-25T11:20:59.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.156+0000 7fb17159a640 1 -- 172.21.15.136:0/1971491608 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c001850 con 0x7fb174070870 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070870 msgr2=0x7fb1740c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070870 0x7fb1740c6960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1680097c0 tx=0x7fb16801ebb0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- 172.21.15.136:0/1971491608 shutdown_connections 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070870 0x7fb1740c6960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- 172.21.15.136:0/1971491608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174143170 0x7fb17415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- 172.21.15.136:0/1971491608 >> 172.21.15.136:0/1971491608 conn(0x7fb1740c9d50 msgr2=0x7fb1740ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- 172.21.15.136:0/1971491608 shutdown_connections 2024-04-25T11:20:59.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- 172.21.15.136:0/1971491608 wait complete. 2024-04-25T11:20:59.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 Processor -- start 2024-04-25T11:20:59.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- start start 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174070870 0x7fb17415a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070c80 0x7fb1740739f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb17415e2e0 con 0x7fb174143170 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb174070350 con 0x7fb174070c80 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb178dcf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb1740712a0 con 0x7fb174070870 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35280/0 (socket says 172.21.15.136:35280) 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 -- 172.21.15.136:0/3401456681 learned_addr learned my addr 172.21.15.136:0/3401456681 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb17259c640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174070870 0x7fb17415a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb171d9b640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070c80 0x7fb1740739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 -- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174070870 msgr2=0x7fb17415a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174070870 0x7fb17415a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 -- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070c80 msgr2=0x7fb1740739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070c80 0x7fb1740739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.160+0000 7fb172d9d640 1 -- 172.21.15.136:0/3401456681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17415a580 con 0x7fb174143170 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb172d9d640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00c530 tx=0x7fb16c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c005cf0 con 0x7fb174143170 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1741f2cc0 con 0x7fb174143170 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16c007070 con 0x7fb174143170 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c007460 con 0x7fb174143170 2024-04-25T11:20:59.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1741f3260 con 0x7fb174143170 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1741599a0 con 0x7fb174143170 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb16c007600 con 0x7fb174143170 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 0x7fb154071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7fb16c08f4f0 con 0x7fb174143170 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb17259c640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 0x7fb154071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:20:59.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.164+0000 7fb17259c640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 0x7fb154071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb164007440 tx=0x7fb16400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:20:59.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.168+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb174143580 con 0x7fb174143170 2024-04-25T11:20:59.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.348+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb17406a350 con 0x7fb174143170 2024-04-25T11:20:59.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb162ffd640 1 -- 172.21.15.136:0/3401456681 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fb16c05bf60 con 0x7fb174143170 2024-04-25T11:20:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:20:59.349 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T11:20:59.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 msgr2=0x7fb154071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 0x7fb154071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb164007440 tx=0x7fb16400b040 comp rx=0 tx=0).stop 2024-04-25T11:20:59.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 msgr2=0x7fb174138b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00c530 tx=0x7fb16c00c9f0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 shutdown_connections 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb174143170 0x7fb174138b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb174070c80 0x7fb1740739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb15406f400 0x7fb154071910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 --2- 172.21.15.136:0/3401456681 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb174070870 0x7fb17415a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 >> 172.21.15.136:0/3401456681 conn(0x7fb1740c9d50 msgr2=0x7fb17413e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 shutdown_connections 2024-04-25T11:20:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:20:59.352+0000 7fb178dcf640 1 -- 172.21.15.136:0/3401456681 wait complete. 2024-04-25T11:21:02.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T11:21:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.528+0000 7fe658cf6640 1 Processor -- start 2024-04-25T11:21:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.528+0000 7fe658cf6640 1 -- start start 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe658cf6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe654156330 0x7fe65415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe658cf6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe65415d010 con 0x7fe6540b5ba0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe658cf6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6540ccb80 con 0x7fe6540cdee0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe658cf6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe6541602f0 con 0x7fe654156330 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe6540b5ba0 0x7fe6541669d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49780/0 (socket says 172.21.15.136:49780) 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 -- 172.21.15.136:0/2802958837 learned_addr learned my addr 172.21.15.136:0/2802958837 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe65259c640 1 --2- 172.21.15.136:0/2802958837 >> v2:172.21.15.157:3300/0 conn(0x7fe654156330 0x7fe65415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308731349 0 0) 0x7fe64c0014a0 con 0x7fe6540b5ba0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe640003530 con 0x7fe6540b5ba0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991687222 0 0) 0x7fe640003530 con 0x7fe6540b5ba0 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 >> v2:172.21.15.157:3300/0 conn(0x7fe654156330 msgr2=0x7fe65415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 --2- 172.21.15.136:0/2802958837 >> v2:172.21.15.157:3300/0 conn(0x7fe654156330 0x7fe65415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 >> v1:172.21.15.157:6789/0 conn(0x7fe6540cdee0 legacy=0x7fe6540b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe654156740 con 0x7fe6540b5ba0 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fe64c001bd0 con 0x7fe6540b5ba0 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 >> v1:172.21.15.136:6789/0 conn(0x7fe6540b5ba0 legacy=0x7fe6541669d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6400039c0 0x7fe640003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe640004810 0x7fe640004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe64000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe6541602f0 con 0x7fe640005160 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe64000c020 con 0x7fe640004810 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe64000c220 con 0x7fe6400039c0 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe64000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe65259c640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6400039c0 0x7fe640003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe64bfff640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe640004810 0x7fe640004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 -- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6400039c0 msgr2=0x7fe640003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe6400039c0 0x7fe640003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 -- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe640004810 msgr2=0x7fe640004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe640004810 0x7fe640004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.532+0000 7fe652d9d640 1 -- 172.21.15.136:0/2802958837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe640003530 con 0x7fe640005160 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe652d9d640 1 -- 172.21.15.136:0/2802958837 reap_dead start 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe652d9d640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe64000b9d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00aa70 tx=0x7fe64c01d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52afc641d978c0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c00a690 con 0x7fe640005160 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c007310 con 0x7fe640005160 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe651d9b640 1 -- 172.21.15.136:0/2802958837 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c007700 con 0x7fe640005160 2024-04-25T11:21:02.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 -- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 msgr2=0x7fe64000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe64000b9d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00aa70 tx=0x7fe64c01d3a0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 -- 172.21.15.136:0/2802958837 shutdown_connections 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2802958837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe64000b9d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 -- 172.21.15.136:0/2802958837 >> 172.21.15.136:0/2802958837 conn(0x7fe65406d6c0 msgr2=0x7fe65406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 -- 172.21.15.136:0/2802958837 shutdown_connections 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.536+0000 7fe658cf6640 1 -- 172.21.15.136:0/2802958837 wait complete. 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 Processor -- start 2024-04-25T11:21:02.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- start start 2024-04-25T11:21:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe654154710 0x7fe6541e5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe654154b20 0x7fe6541e5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe65415fca0 con 0x7fe640005160 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe6540cc9c0 con 0x7fe654154710 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe65415ff80 con 0x7fe654154b20 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35300/0 (socket says 172.21.15.136:35300) 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 -- 172.21.15.136:0/2087167929 learned_addr learned my addr 172.21.15.136:0/2087167929 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe652d9d640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe654154b20 0x7fe6541e5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe64bfff640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe654154710 0x7fe6541e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 -- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe654154b20 msgr2=0x7fe6541e5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe654154b20 0x7fe6541e5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 -- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe654154710 msgr2=0x7fe6541e5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe654154710 0x7fe6541e5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 -- 172.21.15.136:0/2087167929 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6541e7550 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe65259c640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe644011990 tx=0x7fe644011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64400fad0 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64400fc70 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe644014440 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe654157d80 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6541e7af0 con 0x7fe640005160 2024-04-25T11:21:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.540+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65406a350 con 0x7fe640005160 2024-04-25T11:21:02.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.544+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe644012040 con 0x7fe640005160 2024-04-25T11:21:02.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.544+0000 7fe64affd640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 0x7fe634071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:02.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.544+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe64408fca0 con 0x7fe640005160 2024-04-25T11:21:02.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.544+0000 7fe64bfff640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 0x7fe634071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:02.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.544+0000 7fe64bfff640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 0x7fe634071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cbf50 tx=0x7fe63c007bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:02.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.548+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe644019030 con 0x7fe640005160 2024-04-25T11:21:02.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe6540cd9c0 con 0x7fe640005160 2024-04-25T11:21:02.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe64affd640 1 -- 172.21.15.136:0/2087167929 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fe64405c790 con 0x7fe640005160 2024-04-25T11:21:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:02.741 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 msgr2=0x7fe634071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 0x7fe634071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cbf50 tx=0x7fe63c007bc0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 msgr2=0x7fe654158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe644011990 tx=0x7fe644011e50 comp rx=0 tx=0).stop 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 shutdown_connections 2024-04-25T11:21:02.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe63406f430 0x7fe634071940 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe654154b20 0x7fe6541e5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe654154710 0x7fe6541e5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 --2- 172.21.15.136:0/2087167929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe640005160 0x7fe654158840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 >> 172.21.15.136:0/2087167929 conn(0x7fe65406d6c0 msgr2=0x7fe654153110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.744+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 shutdown_connections 2024-04-25T11:21:02.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:02.748+0000 7fe658cf6640 1 -- 172.21.15.136:0/2087167929 wait complete. 2024-04-25T11:21:03.460 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:21:03.462+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:21:05.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T11:21:05.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.924+0000 7f2be9da3640 1 Processor -- start 2024-04-25T11:21:05.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.924+0000 7f2be9da3640 1 -- start start 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be9da3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 0x7f2be40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be9da3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2be415a430 0x7f2be415a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be9da3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be415d750 con 0x7f2be40b5060 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be9da3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2be415da30 con 0x7f2be415a430 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be9da3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2be415dd10 con 0x7f2be40ca290 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2be40b5060 0x7f2be40c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40824/0 (socket says 172.21.15.136:40824) 2024-04-25T11:21:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be37fe640 1 -- 172.21.15.136:0/1359041386 learned_addr learned my addr 172.21.15.136:0/1359041386 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be3fff640 1 --2- 172.21.15.136:0/1359041386 >> v2:172.21.15.136:3300/0 conn(0x7f2be415a430 0x7f2be415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be2ffd640 1 --2- 172.21.15.136:0/1359041386 >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 0x7f2be40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be27fc640 1 -- 172.21.15.136:0/1359041386 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001813859 0 0) 0x7f2be415d750 con 0x7f2be40b5060 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be27fc640 1 -- 172.21.15.136:0/1359041386 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bcc003530 con 0x7f2be40b5060 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be3fff640 1 -- 172.21.15.136:0/1359041386 >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 msgr2=0x7f2be40ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be3fff640 1 --2- 172.21.15.136:0/1359041386 >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 0x7f2be40ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be3fff640 1 -- 172.21.15.136:0/1359041386 >> v1:172.21.15.136:6789/0 conn(0x7f2be40b5060 legacy=0x7f2be40c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be3fff640 1 -- 172.21.15.136:0/1359041386 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be415dff0 con 0x7f2be415a430 2024-04-25T11:21:05.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.928+0000 7f2be2ffd640 1 --2- 172.21.15.136:0/1359041386 >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 0x7f2be40ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be3fff640 1 --2- 172.21.15.136:0/1359041386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be415a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2be40d00d0 tx=0x7f2bd401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ada5f7801fc6b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be27fc640 1 -- 172.21.15.136:0/1359041386 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40b5070 con 0x7f2be415a430 2024-04-25T11:21:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be27fc640 1 -- 172.21.15.136:0/1359041386 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd40a8970 con 0x7f2be415a430 2024-04-25T11:21:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be27fc640 1 -- 172.21.15.136:0/1359041386 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40a8d40 con 0x7f2be415a430 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- 172.21.15.136:0/1359041386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 msgr2=0x7f2be415a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 --2- 172.21.15.136:0/1359041386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be415a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2be40d00d0 tx=0x7f2bd401ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- 172.21.15.136:0/1359041386 shutdown_connections 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 --2- 172.21.15.136:0/1359041386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be415a840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 --2- 172.21.15.136:0/1359041386 >> v2:172.21.15.157:3300/0 conn(0x7f2be40ca290 0x7f2be40ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- 172.21.15.136:0/1359041386 >> 172.21.15.136:0/1359041386 conn(0x7f2be4070f70 msgr2=0x7f2be4068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- 172.21.15.136:0/1359041386 shutdown_connections 2024-04-25T11:21:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- 172.21.15.136:0/1359041386 wait complete. 2024-04-25T11:21:05.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 Processor -- start 2024-04-25T11:21:05.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.932+0000 7f2be9da3640 1 -- start start 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 0x7f2be415bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 0x7f2be4158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2be415ad80 con 0x7f2be415a430 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4159f10 con 0x7f2be40b5060 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2be415d3f0 con 0x7f2be40ca290 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35908/0 (socket says 172.21.15.136:35908) 2024-04-25T11:21:05.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 -- 172.21.15.136:0/3268567465 learned_addr learned my addr 172.21.15.136:0/3268567465 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be2ffd640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 0x7f2be4158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be37fe640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 0x7f2be415bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 -- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 msgr2=0x7f2be4158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 0x7f2be4158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 -- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 msgr2=0x7f2be415bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 0x7f2be415bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 -- 172.21.15.136:0/3268567465 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be415c230 con 0x7f2be415a430 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be2ffd640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 0x7f2be4158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be37fe640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 0x7f2be415bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be3fff640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009880 tx=0x7f2bd401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40bf070 con 0x7f2be415a430 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd40b0640 con 0x7f2be415a430 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41f4370 con 0x7f2be415a430 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40b5040 con 0x7f2be415a430 2024-04-25T11:21:05.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be415b4f0 con 0x7f2be415a430 2024-04-25T11:21:05.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be41f40e0 con 0x7f2be415a430 2024-04-25T11:21:05.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2bd40c2070 con 0x7f2be415a430 2024-04-25T11:21:05.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 0x7f2bd0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:05.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2bd412bed0 con 0x7f2be415a430 2024-04-25T11:21:05.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.936+0000 7f2be37fe640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 0x7f2bd0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:05.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.940+0000 7f2be37fe640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 0x7f2bd0071940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc005c10 tx=0x7f2bdc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:05.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:05.944+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2bd40187d0 con 0x7f2be415a430 2024-04-25T11:21:06.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.148+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2be406a350 con 0x7f2be415a430 2024-04-25T11:21:06.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.148+0000 7f2bcbfff640 1 -- 172.21.15.136:0/3268567465 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f2bd40f8a60 con 0x7f2be415a430 2024-04-25T11:21:06.146 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:06.146 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T11:21:06.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 msgr2=0x7f2bd0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 0x7f2bd0071940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc005c10 tx=0x7f2bdc009040 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 msgr2=0x7f2be41591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009880 tx=0x7f2bd401ba10 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 shutdown_connections 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2be415a430 0x7f2be41591c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2bd006f430 0x7f2bd0071940 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2be40ca290 0x7f2be4158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 --2- 172.21.15.136:0/3268567465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2be40b5060 0x7f2be415bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 >> 172.21.15.136:0/3268567465 conn(0x7f2be4070f70 msgr2=0x7f2be40b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 shutdown_connections 2024-04-25T11:21:06.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:06.152+0000 7f2be9da3640 1 -- 172.21.15.136:0/3268567465 wait complete. 2024-04-25T11:21:08.460 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:21:08.461+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:21:09.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T11:21:09.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.327+0000 7f9a5c26f640 1 Processor -- start 2024-04-25T11:21:09.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.327+0000 7f9a5c26f640 1 -- start start 2024-04-25T11:21:09.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5c26f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a54158da0 0x7f9a541591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5c26f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9a54161650 0x7f9a54163a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5c26f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a541596f0 con 0x7f9a54153190 2024-04-25T11:21:09.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5c26f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9a540c8370 con 0x7f9a54158da0 2024-04-25T11:21:09.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5c26f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9a540c8090 con 0x7f9a54161650 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a54158da0 0x7f9a541591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a54158da0 0x7f9a541591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35920/0 (socket says 172.21.15.136:35920) 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 -- 172.21.15.136:0/2824498455 learned_addr learned my addr 172.21.15.136:0/2824498455 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5a80c640 1 --2- 172.21.15.136:0/2824498455 >> v2:172.21.15.157:3300/0 conn(0x7f9a54161650 0x7f9a54163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 -- 172.21.15.136:0/2824498455 >> v2:172.21.15.157:3300/0 conn(0x7f9a54161650 msgr2=0x7f9a54163a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 --2- 172.21.15.136:0/2824498455 >> v2:172.21.15.157:3300/0 conn(0x7f9a54161650 0x7f9a54163a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 -- 172.21.15.136:0/2824498455 >> v1:172.21.15.157:6789/0 conn(0x7f9a54153190 legacy=0x7f9a540c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.331+0000 7f9a5980a640 1 -- 172.21.15.136:0/2824498455 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a540c6aa0 con 0x7f9a54158da0 2024-04-25T11:21:09.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5980a640 1 --2- 172.21.15.136:0/2824498455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54158da0 0x7f9a541591b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9a54075070 tx=0x7f9a4801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26ab2e96c79cad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a59009640 1 -- 172.21.15.136:0/2824498455 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480b5070 con 0x7f9a54158da0 2024-04-25T11:21:09.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a59009640 1 -- 172.21.15.136:0/2824498455 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a480a8a10 con 0x7f9a54158da0 2024-04-25T11:21:09.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a59009640 1 -- 172.21.15.136:0/2824498455 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a480a8de0 con 0x7f9a54158da0 2024-04-25T11:21:09.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- 172.21.15.136:0/2824498455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54158da0 msgr2=0x7f9a541591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/2824498455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54158da0 0x7f9a541591b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9a54075070 tx=0x7f9a4801ed10 comp rx=0 tx=0).stop 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- 172.21.15.136:0/2824498455 shutdown_connections 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/2824498455 >> v2:172.21.15.157:3300/0 conn(0x7f9a54161650 0x7f9a54163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/2824498455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54158da0 0x7f9a541591b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- 172.21.15.136:0/2824498455 >> 172.21.15.136:0/2824498455 conn(0x7f9a540c9d50 msgr2=0x7f9a540ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- 172.21.15.136:0/2824498455 shutdown_connections 2024-04-25T11:21:09.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- 172.21.15.136:0/2824498455 wait complete. 2024-04-25T11:21:09.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 Processor -- start 2024-04-25T11:21:09.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.335+0000 7f9a5c26f640 1 -- start start 2024-04-25T11:21:09.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a54153190 0x7f9a541f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a54158da0 0x7f9a54143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a540c7340 con 0x7f9a54161650 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a540c7060 con 0x7f9a54158da0 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a540ca870 con 0x7f9a54153190 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35932/0 (socket says 172.21.15.136:35932) 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 -- 172.21.15.136:0/1756620186 learned_addr learned my addr 172.21.15.136:0/1756620186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5980a640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a54158da0 0x7f9a54143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a00b640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a54153190 0x7f9a541f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 -- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a54153190 msgr2=0x7f9a541f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a54153190 0x7f9a541f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 -- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a54158da0 msgr2=0x7f9a54143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a54158da0 0x7f9a54143c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 -- 172.21.15.136:0/1756620186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a541f6a10 con 0x7f9a54161650 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a80c640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c007e90 tx=0x7f9a4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c014420 con 0x7f9a54161650 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c019070 con 0x7f9a54161650 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c014800 con 0x7f9a54161650 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a541f2d50 con 0x7f9a54161650 2024-04-25T11:21:09.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a541f3010 con 0x7f9a54161650 2024-04-25T11:21:09.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5406a350 con 0x7f9a54161650 2024-04-25T11:21:09.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a4c0149a0 con 0x7f9a54161650 2024-04-25T11:21:09.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 0x7f9a40071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9a4c08ff70 con 0x7f9a54161650 2024-04-25T11:21:09.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.339+0000 7f9a5a00b640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 0x7f9a40071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.343+0000 7f9a5a00b640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 0x7f9a40071910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a44000f70 tx=0x7f9a44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.347+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a4c05ca60 con 0x7f9a54161650 2024-04-25T11:21:09.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.535+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9a540699a0 con 0x7f9a54161650 2024-04-25T11:21:09.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.535+0000 7f9a52ffd640 1 -- 172.21.15.136:0/1756620186 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f9a4c05c360 con 0x7f9a54161650 2024-04-25T11:21:09.534 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:09.535 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T11:21:09.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 msgr2=0x7f9a40071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 0x7f9a40071910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a44000f70 tx=0x7f9a44009040 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 msgr2=0x7f9a54144160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c007e90 tx=0x7f9a4c00e4b0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 shutdown_connections 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a54161650 0x7f9a54144160 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a54158da0 0x7f9a54143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a4006f400 0x7f9a40071910 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a44000f70 tx=0x7f9a44009040 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 --2- 172.21.15.136:0/1756620186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a54153190 0x7f9a541f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 >> 172.21.15.136:0/1756620186 conn(0x7f9a540c9d50 msgr2=0x7f9a541459f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 shutdown_connections 2024-04-25T11:21:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.539+0000 7f9a5c26f640 1 -- 172.21.15.136:0/1756620186 wait complete. 2024-04-25T11:21:09.554 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T11:21:09.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:09.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.703+0000 7fd69f275640 1 Processor -- start 2024-04-25T11:21:09.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.703+0000 7fd69f275640 1 -- start start 2024-04-25T11:21:09.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69f275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd69815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69f275640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd698143580 con 0x7fd698070870 2024-04-25T11:21:09.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69f275640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69815dd20 con 0x7fd698070c80 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69f275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd69815e000 con 0x7fd698143170 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69c810640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd698070870 0x7fd6980c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40830/0 (socket says 172.21.15.136:40830) 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69c810640 1 -- 172.21.15.136:0/1765922248 learned_addr learned my addr 172.21.15.136:0/1765922248 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d011640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd69815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 610812399 0 0) 0x7fd688001120 con 0x7fd698070870 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd684003530 con 0x7fd698070870 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669438757 0 0) 0x7fd69815dd20 con 0x7fd698070c80 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd688001120 con 0x7fd698070c80 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1467737599 0 0) 0x7fd684003530 con 0x7fd698070870 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 msgr2=0x7fd69815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd69815d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 >> v1:172.21.15.157:6789/0 conn(0x7fd698070c80 legacy=0x7fd698164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69806fd90 con 0x7fd698070870 2024-04-25T11:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd688001850 con 0x7fd698070870 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 >> v1:172.21.15.136:6789/0 conn(0x7fd698070870 legacy=0x7fd6980c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd684003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6840040e0 0x7fd684004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd684004d50 0x7fd684005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd69815e000 con 0x7fd684004d50 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd68400bb30 con 0x7fd6840040e0 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd68400bd30 con 0x7fd698143170 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd684004d50 0x7fd684005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69c810640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6840040e0 0x7fd684004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d011640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd684003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 -- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 msgr2=0x7fd684003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:09.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd684003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 -- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6840040e0 msgr2=0x7fd684004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:09.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6840040e0 0x7fd684004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d812640 1 -- 172.21.15.136:0/1765922248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd684003530 con 0x7fd684004d50 2024-04-25T11:21:09.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.707+0000 7fd69d011640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698143170 0x7fd684003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:09.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69d812640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd684004d50 0x7fd684005160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6900018d0 tx=0x7fd69001c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acaf1c2d8616af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6900af070 con 0x7fd684004d50 2024-04-25T11:21:09.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6900052f0 con 0x7fd684004d50 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd68ffff640 1 -- 172.21.15.136:0/1765922248 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690001520 con 0x7fd684004d50 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- 172.21.15.136:0/1765922248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd684004d50 msgr2=0x7fd684005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 --2- 172.21.15.136:0/1765922248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd684004d50 0x7fd684005160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6900018d0 tx=0x7fd69001c580 comp rx=0 tx=0).stop 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69d812640 1 -- 172.21.15.136:0/1765922248 reap_dead start 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- 172.21.15.136:0/1765922248 shutdown_connections 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- 172.21.15.136:0/1765922248 >> 172.21.15.136:0/1765922248 conn(0x7fd6980c9d50 msgr2=0x7fd6980ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- 172.21.15.136:0/1765922248 shutdown_connections 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- 172.21.15.136:0/1765922248 wait complete. 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 Processor -- start 2024-04-25T11:21:09.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.711+0000 7fd69f275640 1 -- start start 2024-04-25T11:21:09.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 0x7fd6981567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698156ce0 0x7fd69813ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd69815e2e0 con 0x7fd6981586d0 2024-04-25T11:21:09.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd698070350 con 0x7fd6981e57e0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6980712a0 con 0x7fd698156ce0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35952/0 (socket says 172.21.15.136:35952) 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 -- 172.21.15.136:0/844800235 learned_addr learned my addr 172.21.15.136:0/844800235 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d812640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698156ce0 0x7fd69813ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69c810640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 0x7fd6981567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 -- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698156ce0 msgr2=0x7fd69813ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698156ce0 0x7fd69813ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 -- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 msgr2=0x7fd6981567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 0x7fd6981567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 -- 172.21.15.136:0/844800235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6980c8560 con 0x7fd6981586d0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69c810640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 0x7fd6981567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69d011640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd69400f380 tx=0x7fd69400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694003ba0 con 0x7fd6981586d0 2024-04-25T11:21:09.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd694003d40 con 0x7fd6981586d0 2024-04-25T11:21:09.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6981f41d0 con 0x7fd6981586d0 2024-04-25T11:21:09.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694004220 con 0x7fd6981586d0 2024-04-25T11:21:09.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6981f3410 con 0x7fd6981586d0 2024-04-25T11:21:09.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69806a350 con 0x7fd6981586d0 2024-04-25T11:21:09.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.715+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd69401e020 con 0x7fd6981586d0 2024-04-25T11:21:09.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.719+0000 7fd68dffb640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 0x7fd67c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:09.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.719+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd69408fd30 con 0x7fd6981586d0 2024-04-25T11:21:09.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.719+0000 7fd69c810640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 0x7fd67c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:09.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.719+0000 7fd69c810640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 0x7fd67c071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd688007fd0 tx=0x7fd688009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:09.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.723+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd694012070 con 0x7fd6981586d0 2024-04-25T11:21:09.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.883+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd698154540 con 0x7fd6981586d0 2024-04-25T11:21:09.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.887+0000 7fd68dffb640 1 -- 172.21.15.136:0/844800235 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7fd69405c940 con 0x7fd6981586d0 2024-04-25T11:21:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:09.886 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 msgr2=0x7fd67c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 0x7fd67c071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd688007fd0 tx=0x7fd688009040 comp rx=0 tx=0).stop 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 msgr2=0x7fd698155030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd69400f380 tx=0x7fd69400f840 comp rx=0 tx=0).stop 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 shutdown_connections 2024-04-25T11:21:09.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd67c06f400 0x7fd67c071910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd698156ce0 0x7fd69813ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6981e57e0 0x7fd6981567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 --2- 172.21.15.136:0/844800235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6981586d0 0x7fd698155030 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 >> 172.21.15.136:0/844800235 conn(0x7fd6980c9d50 msgr2=0x7fd69813dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 shutdown_connections 2024-04-25T11:21:09.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:09.891+0000 7fd69f275640 1 -- 172.21.15.136:0/844800235 wait complete. 2024-04-25T11:21:09.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:21:09.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:21:10.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 Processor -- start 2024-04-25T11:21:10.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 -- start start 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3500ca870 0x7fc350152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc3500c87d0 0x7fc350162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3500c9a40 con 0x7fc3500c83c0 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc3500c7390 con 0x7fc3500ca870 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc35828b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc350152a60 con 0x7fc3500c87d0 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356027640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3500ca870 0x7fc350152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc355826640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc3500c83c0 0x7fc3500736a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40832/0 (socket says 172.21.15.136:40832) 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc355826640 1 -- 172.21.15.136:0/1522346752 learned_addr learned my addr 172.21.15.136:0/1522346752 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356828640 1 --2- 172.21.15.136:0/1522346752 >> v2:172.21.15.157:3300/0 conn(0x7fc3500c87d0 0x7fc350162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356027640 1 -- 172.21.15.136:0/1522346752 >> v2:172.21.15.157:3300/0 conn(0x7fc3500c87d0 msgr2=0x7fc350162cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356027640 1 --2- 172.21.15.136:0/1522346752 >> v2:172.21.15.157:3300/0 conn(0x7fc3500c87d0 0x7fc350162cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356027640 1 -- 172.21.15.136:0/1522346752 >> v1:172.21.15.136:6789/0 conn(0x7fc3500c83c0 legacy=0x7fc3500736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc355025640 1 -- 172.21.15.136:0/1522346752 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026302120 0 0) 0x7fc3500c9a40 con 0x7fc3500c83c0 2024-04-25T11:21:10.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.059+0000 7fc356027640 1 -- 172.21.15.136:0/1522346752 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3500c9480 con 0x7fc3500ca870 2024-04-25T11:21:10.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc356027640 1 --2- 172.21.15.136:0/1522346752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500ca870 0x7fc350152520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc350074f20 tx=0x7fc34c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ddd692c4ef4d93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc355025640 1 -- 172.21.15.136:0/1522346752 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0b1070 con 0x7fc3500ca870 2024-04-25T11:21:10.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc355025640 1 -- 172.21.15.136:0/1522346752 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c0ac640 con 0x7fc3500ca870 2024-04-25T11:21:10.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc355025640 1 -- 172.21.15.136:0/1522346752 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0aca10 con 0x7fc3500ca870 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 -- 172.21.15.136:0/1522346752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500ca870 msgr2=0x7fc350152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 --2- 172.21.15.136:0/1522346752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500ca870 0x7fc350152520 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc350074f20 tx=0x7fc34c01c480 comp rx=0 tx=0).stop 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 -- 172.21.15.136:0/1522346752 shutdown_connections 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 --2- 172.21.15.136:0/1522346752 >> v2:172.21.15.157:3300/0 conn(0x7fc3500c87d0 0x7fc350162cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 --2- 172.21.15.136:0/1522346752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500ca870 0x7fc350152520 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 -- 172.21.15.136:0/1522346752 >> 172.21.15.136:0/1522346752 conn(0x7fc3500c9d50 msgr2=0x7fc3500ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 -- 172.21.15.136:0/1522346752 shutdown_connections 2024-04-25T11:21:10.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.063+0000 7fc35828b640 1 -- 172.21.15.136:0/1522346752 wait complete. 2024-04-25T11:21:10.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 Processor -- start 2024-04-25T11:21:10.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- start start 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3500c83c0 0x7fc3501f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3500ca870 0x7fc3501439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc350073f30 con 0x7fc3500c87d0 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc350073db0 con 0x7fc3500c83c0 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc3500c4750 con 0x7fc3500ca870 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35982/0 (socket says 172.21.15.136:35982) 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 -- 172.21.15.136:0/1082096430 learned_addr learned my addr 172.21.15.136:0/1082096430 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc356027640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3500c83c0 0x7fc3501f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc356828640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3500ca870 0x7fc3501439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 -- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3500ca870 msgr2=0x7fc3501439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3500ca870 0x7fc3501439f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 -- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3500c83c0 msgr2=0x7fc3501f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3500c83c0 0x7fc3501f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 -- 172.21.15.136:0/1082096430 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35014c860 con 0x7fc3500c87d0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc355826640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c450 tx=0x7fc34000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400074e0 con 0x7fc3500c87d0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3400047f0 con 0x7fc3500c87d0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3501f3e60 con 0x7fc3500c87d0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340007680 con 0x7fc3500c87d0 2024-04-25T11:21:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc350156a20 con 0x7fc3500c87d0 2024-04-25T11:21:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.067+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35006a350 con 0x7fc3500c87d0 2024-04-25T11:21:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.071+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc340011070 con 0x7fc3500c87d0 2024-04-25T11:21:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.071+0000 7fc346ffd640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 0x7fc334071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.071+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc34008f210 con 0x7fc3500c87d0 2024-04-25T11:21:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.071+0000 7fc356027640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 0x7fc334071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.071+0000 7fc356027640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 0x7fc334071940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0ae000 tx=0x7fc34c0a6070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.075+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc34005bd00 con 0x7fc3500c87d0 2024-04-25T11:21:10.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc3500c7ce0 con 0x7fc3500c87d0 2024-04-25T11:21:10.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc346ffd640 1 -- 172.21.15.136:0/1082096430 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fc34005b600 con 0x7fc3500c87d0 2024-04-25T11:21:10.222 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 msgr2=0x7fc334071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 0x7fc334071940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0ae000 tx=0x7fc34c0a6070 comp rx=0 tx=0).stop 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 msgr2=0x7fc350148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c450 tx=0x7fc34000c910 comp rx=0 tx=0).stop 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 shutdown_connections 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33406f430 0x7fc334071940 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc3500ca870 0x7fc3501439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3500c87d0 0x7fc350148f80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 --2- 172.21.15.136:0/1082096430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3500c83c0 0x7fc3501f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 >> 172.21.15.136:0/1082096430 conn(0x7fc3500c9d50 msgr2=0x7fc350151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 shutdown_connections 2024-04-25T11:21:10.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.223+0000 7fc35828b640 1 -- 172.21.15.136:0/1082096430 wait complete. 2024-04-25T11:21:10.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:21:10.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.391+0000 7fb1eff06640 1 Processor -- start 2024-04-25T11:21:10.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.391+0000 7fb1eff06640 1 -- start start 2024-04-25T11:21:10.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1eff06640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb1e8152520 0x7fb1e80736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1eff06640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb1e8073be0 0x7fb1e8162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1eff06640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80c9a40 con 0x7fb1e80ca870 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1eff06640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e80c7390 con 0x7fb1e8152520 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1eff06640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e80c9480 con 0x7fb1e8073be0 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb1e8152520 0x7fb1e80736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb1e8152520 0x7fb1e80736a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35996/0 (socket says 172.21.15.136:35996) 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 -- 172.21.15.136:0/2978643376 learned_addr learned my addr 172.21.15.136:0/2978643376 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ee4a3640 1 --2- 172.21.15.136:0/2978643376 >> v2:172.21.15.157:3300/0 conn(0x7fb1e8073be0 0x7fb1e8162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 -- 172.21.15.136:0/2978643376 >> v2:172.21.15.157:3300/0 conn(0x7fb1e8073be0 msgr2=0x7fb1e8162cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 --2- 172.21.15.136:0/2978643376 >> v2:172.21.15.157:3300/0 conn(0x7fb1e8073be0 0x7fb1e8162cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 -- 172.21.15.136:0/2978643376 >> v1:172.21.15.157:6789/0 conn(0x7fb1e80ca870 legacy=0x7fb1e80c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 -- 172.21.15.136:0/2978643376 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e80c9760 con 0x7fb1e8152520 2024-04-25T11:21:10.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.395+0000 7fb1ed4a1640 1 --2- 172.21.15.136:0/2978643376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8152520 0x7fb1e80736a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8074f20 tx=0x7fb1d801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424b435c2d3ad731 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1ecca0640 1 -- 172.21.15.136:0/2978643376 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80b0070 con 0x7fb1e8152520 2024-04-25T11:21:10.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1ecca0640 1 -- 172.21.15.136:0/2978643376 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d80ab600 con 0x7fb1e8152520 2024-04-25T11:21:10.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1ecca0640 1 -- 172.21.15.136:0/2978643376 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80ab9d0 con 0x7fb1e8152520 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- 172.21.15.136:0/2978643376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8152520 msgr2=0x7fb1e80736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2978643376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8152520 0x7fb1e80736a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8074f20 tx=0x7fb1d801c520 comp rx=0 tx=0).stop 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- 172.21.15.136:0/2978643376 shutdown_connections 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2978643376 >> v2:172.21.15.157:3300/0 conn(0x7fb1e8073be0 0x7fb1e8162cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2978643376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8152520 0x7fb1e80736a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- 172.21.15.136:0/2978643376 >> 172.21.15.136:0/2978643376 conn(0x7fb1e80c9d50 msgr2=0x7fb1e80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- 172.21.15.136:0/2978643376 shutdown_connections 2024-04-25T11:21:10.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- 172.21.15.136:0/2978643376 wait complete. 2024-04-25T11:21:10.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 Processor -- start 2024-04-25T11:21:10.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.399+0000 7fb1eff06640 1 -- start start 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1e80ca870 0x7fb1e814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1e8152520 0x7fb1e8143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e80c48d0 con 0x7fb1e8073be0 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e80c4750 con 0x7fb1e8152520 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e8074250 con 0x7fb1e80ca870 2024-04-25T11:21:10.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36008/0 (socket says 172.21.15.136:36008) 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 -- 172.21.15.136:0/2824737410 learned_addr learned my addr 172.21.15.136:0/2824737410 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1ee4a3640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1e8152520 0x7fb1e8143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1ed4a1640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1e80ca870 0x7fb1e814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 -- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1e80ca870 msgr2=0x7fb1e814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1e80ca870 0x7fb1e814b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 -- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1e8152520 msgr2=0x7fb1e8143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1e8152520 0x7fb1e8143af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 -- 172.21.15.136:0/2824737410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e81f71d0 con 0x7fb1e8073be0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1edca2640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c370 tx=0x7fb1e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40052d0 con 0x7fb1e8073be0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e4005470 con 0x7fb1e8073be0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e8153940 con 0x7fb1e8073be0 2024-04-25T11:21:10.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4007260 con 0x7fb1e8073be0 2024-04-25T11:21:10.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e8157490 con 0x7fb1e8073be0 2024-04-25T11:21:10.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e806a350 con 0x7fb1e8073be0 2024-04-25T11:21:10.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb1e4007400 con 0x7fb1e8073be0 2024-04-25T11:21:10.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 0x7fb1cc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb1e408ea10 con 0x7fb1e8073be0 2024-04-25T11:21:10.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.403+0000 7fb1ed4a1640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 0x7fb1cc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.407+0000 7fb1ed4a1640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 0x7fb1cc071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80ad000 tx=0x7fb1d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.411+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb1e405b620 con 0x7fb1e8073be0 2024-04-25T11:21:10.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.559+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb1e8071350 con 0x7fb1e8073be0 2024-04-25T11:21:10.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.559+0000 7fb1de7fc640 1 -- 172.21.15.136:0/2824737410 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fb1e405af20 con 0x7fb1e8073be0 2024-04-25T11:21:10.559 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 msgr2=0x7fb1cc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 0x7fb1cc071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80ad000 tx=0x7fb1d8008040 comp rx=0 tx=0).stop 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 msgr2=0x7fb1e81f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400c370 tx=0x7fb1e400c830 comp rx=0 tx=0).stop 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 shutdown_connections 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1e8152520 0x7fb1e8143af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb1cc06f430 0x7fb1cc071940 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb1e80ca870 0x7fb1e814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 --2- 172.21.15.136:0/2824737410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1e8073be0 0x7fb1e81f6c90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 >> 172.21.15.136:0/2824737410 conn(0x7fb1e80c9d50 msgr2=0x7fb1e80cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 shutdown_connections 2024-04-25T11:21:10.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.563+0000 7fb1eff06640 1 -- 172.21.15.136:0/2824737410 wait complete. 2024-04-25T11:21:10.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:21:10.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 Processor -- start 2024-04-25T11:21:10.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 -- start start 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fecd00c8e40 0x7fecd00c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd0159900 con 0x7fecd00c4c90 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fecd01596d0 con 0x7fecd00c8e40 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecd5c6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fecd0159bc0 con 0x7fecd00c7c50 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7fecc7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fecd00c8e40 0x7fecd00c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45214/0 (socket says 172.21.15.136:45214) 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 -- 172.21.15.136:0/2556289264 learned_addr learned my addr 172.21.15.136:0/2556289264 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccffff640 1 --1- 172.21.15.136:0/2556289264 >> v1:172.21.15.157:6789/0 conn(0x7fecd00c4c90 0x7fecd00c68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:41682/0 (socket says 172.21.15.136:41682) 2024-04-25T11:21:10.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 -- 172.21.15.136:0/2556289264 >> v2:172.21.15.157:3300/0 conn(0x7fecd00c8e40 msgr2=0x7fecd00c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:10.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 --2- 172.21.15.136:0/2556289264 >> v2:172.21.15.157:3300/0 conn(0x7fecd00c8e40 0x7fecd00c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 -- 172.21.15.136:0/2556289264 >> v1:172.21.15.157:6789/0 conn(0x7fecd00c4c90 legacy=0x7fecd00c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.731+0000 7feccf7fe640 1 -- 172.21.15.136:0/2556289264 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd015c3e0 con 0x7fecd00c7c50 2024-04-25T11:21:10.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7feccf7fe640 1 --2- 172.21.15.136:0/2556289264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009870 tx=0x7fecc001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc80369399005154 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecceffd640 1 -- 172.21.15.136:0/2556289264 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc00b5070 con 0x7fecd00c7c50 2024-04-25T11:21:10.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecceffd640 1 -- 172.21.15.136:0/2556289264 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc00a8a50 con 0x7fecd00c7c50 2024-04-25T11:21:10.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecceffd640 1 -- 172.21.15.136:0/2556289264 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc00a8e20 con 0x7fecd00c7c50 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 -- 172.21.15.136:0/2556289264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 msgr2=0x7fecd00c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/2556289264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009870 tx=0x7fecc001ed50 comp rx=0 tx=0).stop 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 -- 172.21.15.136:0/2556289264 shutdown_connections 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/2556289264 >> v2:172.21.15.157:3300/0 conn(0x7fecd00c8e40 0x7fecd00c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/2556289264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c7c50 0x7fecd00c8900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 -- 172.21.15.136:0/2556289264 >> 172.21.15.136:0/2556289264 conn(0x7fecd00c9d50 msgr2=0x7fecd00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 -- 172.21.15.136:0/2556289264 shutdown_connections 2024-04-25T11:21:10.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.735+0000 7fecd5c6f640 1 -- 172.21.15.136:0/2556289264 wait complete. 2024-04-25T11:21:10.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 Processor -- start 2024-04-25T11:21:10.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- start start 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c4c90 0x7fecd01e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecd00c7c50 0x7fecd01e34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fecd015cc80 con 0x7fecd00c8e40 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fecd015c9a0 con 0x7fecd00c7c50 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fecd00c5270 con 0x7fecd00c4c90 2024-04-25T11:21:10.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36020/0 (socket says 172.21.15.136:36020) 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 -- 172.21.15.136:0/1889774762 learned_addr learned my addr 172.21.15.136:0/1889774762 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccf7fe640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c4c90 0x7fecd01e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecc7fff640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecd00c7c50 0x7fecd01e34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 -- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c4c90 msgr2=0x7fecd01e5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c4c90 0x7fecd01e5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 -- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecd00c7c50 msgr2=0x7fecd01e34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecd00c7c50 0x7fecd01e34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 -- 172.21.15.136:0/1889774762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd00ccd50 con 0x7fecd00c8e40 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7feccffff640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fecc800c730 tx=0x7fecc800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8018070 con 0x7fecd00c8e40 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc8004d70 con 0x7fecd00c8e40 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd00cc790 con 0x7fecd00c8e40 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc800a880 con 0x7fecd00c8e40 2024-04-25T11:21:10.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd01e23d0 con 0x7fecd00c8e40 2024-04-25T11:21:10.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.739+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd01e4210 con 0x7fecd00c8e40 2024-04-25T11:21:10.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.743+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fecc801b070 con 0x7fecd00c8e40 2024-04-25T11:21:10.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.743+0000 7fecccff9640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 0x7fecb0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:10.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.743+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fecc808e910 con 0x7fecd00c8e40 2024-04-25T11:21:10.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.743+0000 7feccf7fe640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 0x7fecb0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:10.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.743+0000 7feccf7fe640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 0x7fecb0071940 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fecc00b2000 tx=0x7fecc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:10.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.747+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fecd01e4210 con 0x7fecd00c8e40 2024-04-25T11:21:10.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.895+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fecd006a350 con 0x7fecd00c8e40 2024-04-25T11:21:10.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.895+0000 7fecccff9640 1 -- 172.21.15.136:0/1889774762 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fecc805aed0 con 0x7fecd00c8e40 2024-04-25T11:21:10.896 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 msgr2=0x7fecb0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 0x7fecb0071940 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fecc00b2000 tx=0x7fecc0009040 comp rx=0 tx=0).stop 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 msgr2=0x7fecd01e4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fecc800c730 tx=0x7fecc800cb30 comp rx=0 tx=0).stop 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 shutdown_connections 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd00c8e40 0x7fecd01e4550 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fecd00c7c50 0x7fecd01e34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fecb006f430 0x7fecb0071940 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 --2- 172.21.15.136:0/1889774762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fecd00c4c90 0x7fecd01e5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:10.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 >> 172.21.15.136:0/1889774762 conn(0x7fecd00c9d50 msgr2=0x7fecd014b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:10.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 shutdown_connections 2024-04-25T11:21:10.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:10.899+0000 7fecd5c6f640 1 -- 172.21.15.136:0/1889774762 wait complete. 2024-04-25T11:21:10.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:21:11.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.055+0000 7f012fabd640 1 Processor -- start 2024-04-25T11:21:11.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012fabd640 1 -- start start 2024-04-25T11:21:11.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012fabd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0128072dd0 0x7f01280c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012fabd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012815d590 con 0x7f01280c8190 2024-04-25T11:21:11.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012fabd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012815d870 con 0x7f01280729c0 2024-04-25T11:21:11.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012fabd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f012815d340 con 0x7f0128072dd0 2024-04-25T11:21:11.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f01280c8190 0x7f0128073590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40848/0 (socket says 172.21.15.136:40848) 2024-04-25T11:21:11.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 -- 172.21.15.136:0/3216766368 learned_addr learned my addr 172.21.15.136:0/3216766368 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:11.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012d058640 1 --2- 172.21.15.136:0/3216766368 >> v2:172.21.15.157:3300/0 conn(0x7f0128072dd0 0x7f01280c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047386490 0 0) 0x7f012815d590 con 0x7f01280c8190 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0114003530 con 0x7f01280c8190 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971641923 0 0) 0x7f0114003530 con 0x7f01280c8190 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 >> v2:172.21.15.157:3300/0 conn(0x7f0128072dd0 msgr2=0x7f01280c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 --2- 172.21.15.136:0/3216766368 >> v2:172.21.15.157:3300/0 conn(0x7f0128072dd0 0x7f01280c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 >> v1:172.21.15.157:6789/0 conn(0x7f01280729c0 legacy=0x7f0128142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012815db50 con 0x7f01280c8190 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f0120001a00 con 0x7f01280c8190 2024-04-25T11:21:11.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 >> v1:172.21.15.136:6789/0 conn(0x7f01280c8190 legacy=0x7f0128073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01140039c0 0x7f0114003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0114004810 0x7f0114004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f011400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f012815d340 con 0x7f0114005160 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f011400c0d0 con 0x7f01140039c0 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f011400c2d0 con 0x7f0114004810 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012d058640 1 --2- 172.21.15.136:0/3216766368 >> v2:172.21.15.157:3300/0 conn(0x7f0128072dd0 0x7f01280c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f011400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012d859640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01140039c0 0x7f0114003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012d058640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0114004810 0x7f0114004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 -- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0114004810 msgr2=0x7f0114004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:11.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0114004810 0x7f0114004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 -- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01140039c0 msgr2=0x7f0114003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:11.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01140039c0 0x7f0114003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012e05a640 1 -- 172.21.15.136:0/3216766368 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0114003530 con 0x7f0114005160 2024-04-25T11:21:11.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.059+0000 7f012d859640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01140039c0 0x7f0114003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012e05a640 1 -- 172.21.15.136:0/3216766368 reap_dead start 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012e05a640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f011400b9d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f012000a850 tx=0x7f012001fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f91753e4f21b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012000a4a0 con 0x7f0114005160 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01200aa4f0 con 0x7f0114005160 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012c857640 1 -- 172.21.15.136:0/3216766368 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01200aa8c0 con 0x7f0114005160 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 -- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 msgr2=0x7f011400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f011400b9d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f012000a850 tx=0x7f012001fac0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 -- 172.21.15.136:0/3216766368 shutdown_connections 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 --2- 172.21.15.136:0/3216766368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f011400b9d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 -- 172.21.15.136:0/3216766368 >> 172.21.15.136:0/3216766368 conn(0x7f01280c9d50 msgr2=0x7f01280ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 -- 172.21.15.136:0/3216766368 shutdown_connections 2024-04-25T11:21:11.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.063+0000 7f012fabd640 1 -- 172.21.15.136:0/3216766368 wait complete. 2024-04-25T11:21:11.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 Processor -- start 2024-04-25T11:21:11.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- start start 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01281efab0 0x7f01281e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 0x7f0128135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f01280c7520 con 0x7f0114005160 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f012815e1a0 con 0x7f01281efab0 2024-04-25T11:21:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f01280c76a0 con 0x7f01281f3020 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36042/0 (socket says 172.21.15.136:36042) 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 -- 172.21.15.136:0/199914837 learned_addr learned my addr 172.21.15.136:0/199914837 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d058640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01281efab0 0x7f01281e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012e05a640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 0x7f0128135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 -- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 msgr2=0x7f0128135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 0x7f0128135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 -- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01281efab0 msgr2=0x7f01281e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01281efab0 0x7f01281e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 -- 172.21.15.136:0/199914837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012815c970 con 0x7f0114005160 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012e05a640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 0x7f0128135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012d859640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f012400e4c0 tx=0x7f012400e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240052e0 con 0x7f0114005160 2024-04-25T11:21:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0124018070 con 0x7f0114005160 2024-04-25T11:21:11.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012815b7f0 con 0x7f0114005160 2024-04-25T11:21:11.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124005480 con 0x7f0114005160 2024-04-25T11:21:11.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01281f0620 con 0x7f0114005160 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.067+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f012806a350 con 0x7f0114005160 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.071+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0124003840 con 0x7f0114005160 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.071+0000 7f011e7fc640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 0x7f010c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.071+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f012408f390 con 0x7f0114005160 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.071+0000 7f012d058640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 0x7f010c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.071+0000 7f012d058640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 0x7f010c071940 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0128070dd0 tx=0x7f0118008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.075+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f012405be80 con 0x7f0114005160 2024-04-25T11:21:11.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f01281544c0 con 0x7f0114005160 2024-04-25T11:21:11.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f011e7fc640 1 -- 172.21.15.136:0/199914837 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f012405b780 con 0x7f0114005160 2024-04-25T11:21:11.221 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 msgr2=0x7f010c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 0x7f010c071940 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0128070dd0 tx=0x7f0118008040 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 msgr2=0x7f012815c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f012400e4c0 tx=0x7f012400e980 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 shutdown_connections 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f010c06f430 0x7f010c071940 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01281f3020 0x7f0128135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01281efab0 0x7f01281e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 --2- 172.21.15.136:0/199914837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0114005160 0x7f012815c430 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 >> 172.21.15.136:0/199914837 conn(0x7f01280c9d50 msgr2=0x7f01280c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 shutdown_connections 2024-04-25T11:21:11.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.223+0000 7f012fabd640 1 -- 172.21.15.136:0/199914837 wait complete. 2024-04-25T11:21:11.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:21:11.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 Processor -- start 2024-04-25T11:21:11.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 -- start start 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a040cdee0 0x7f9a040ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9a040b5490 0x7f9a040b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a040ccb80 con 0x7f9a04156330 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9a041602f0 con 0x7f9a040cdee0 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a09805640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9a04156740 con 0x7f9a040b5490 2024-04-25T11:21:11.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a02ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9a04156330 0x7f9a0415cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40858/0 (socket says 172.21.15.136:40858) 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a02ffd640 1 -- 172.21.15.136:0/964320970 learned_addr learned my addr 172.21.15.136:0/964320970 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a027fc640 1 --2- 172.21.15.136:0/964320970 >> v2:172.21.15.136:3300/0 conn(0x7f9a040cdee0 0x7f9a040ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a037fe640 1 --2- 172.21.15.136:0/964320970 >> v2:172.21.15.157:3300/0 conn(0x7f9a040b5490 0x7f9a040b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a01ffb640 1 -- 172.21.15.136:0/964320970 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3067272868 0 0) 0x7f9a040ccb80 con 0x7f9a04156330 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a01ffb640 1 -- 172.21.15.136:0/964320970 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99ec003530 con 0x7f9a04156330 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a027fc640 1 -- 172.21.15.136:0/964320970 >> v2:172.21.15.157:3300/0 conn(0x7f9a040b5490 msgr2=0x7f9a040b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a027fc640 1 --2- 172.21.15.136:0/964320970 >> v2:172.21.15.157:3300/0 conn(0x7f9a040b5490 0x7f9a040b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a027fc640 1 -- 172.21.15.136:0/964320970 >> v1:172.21.15.136:6789/0 conn(0x7f9a04156330 legacy=0x7f9a0415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.379+0000 7f9a027fc640 1 -- 172.21.15.136:0/964320970 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0415f6e0 con 0x7f9a040cdee0 2024-04-25T11:21:11.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a027fc640 1 --2- 172.21.15.136:0/964320970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040cdee0 0x7f9a040ce2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a040b7120 tx=0x7f99f801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d753aa9f8e825df7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a01ffb640 1 -- 172.21.15.136:0/964320970 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80b5070 con 0x7f9a040cdee0 2024-04-25T11:21:11.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a01ffb640 1 -- 172.21.15.136:0/964320970 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f80a8970 con 0x7f9a040cdee0 2024-04-25T11:21:11.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a01ffb640 1 -- 172.21.15.136:0/964320970 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80a8d40 con 0x7f9a040cdee0 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- 172.21.15.136:0/964320970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040cdee0 msgr2=0x7f9a040ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 --2- 172.21.15.136:0/964320970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040cdee0 0x7f9a040ce2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a040b7120 tx=0x7f99f801ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- 172.21.15.136:0/964320970 shutdown_connections 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 --2- 172.21.15.136:0/964320970 >> v2:172.21.15.157:3300/0 conn(0x7f9a040b5490 0x7f9a040b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 --2- 172.21.15.136:0/964320970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040cdee0 0x7f9a040ce2f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- 172.21.15.136:0/964320970 >> 172.21.15.136:0/964320970 conn(0x7f9a0406d6c0 msgr2=0x7f9a0406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- 172.21.15.136:0/964320970 shutdown_connections 2024-04-25T11:21:11.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- 172.21.15.136:0/964320970 wait complete. 2024-04-25T11:21:11.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 Processor -- start 2024-04-25T11:21:11.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.383+0000 7f9a09805640 1 -- start start 2024-04-25T11:21:11.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a040cdee0 0x7f9a041f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 0x7f9a041e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0415ff80 con 0x7f9a040b5490 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0415fca0 con 0x7f9a040cdee0 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a040b5de0 con 0x7f9a04156330 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36064/0 (socket says 172.21.15.136:36064) 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 -- 172.21.15.136:0/2081901655 learned_addr learned my addr 172.21.15.136:0/2081901655 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a027fc640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a040cdee0 0x7f9a041f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a037fe640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 0x7f9a041e78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 -- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 msgr2=0x7f9a041e78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 0x7f9a041e78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 -- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a040cdee0 msgr2=0x7f9a041f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a040cdee0 0x7f9a041f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 -- 172.21.15.136:0/2081901655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a041f2ee0 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a037fe640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 0x7f9a041e78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a02ffd640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c3f0 tx=0x7f99f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4017070 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f4005cf0 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a041f5ed0 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f40072f0 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a041f6490 con 0x7f9a040b5490 2024-04-25T11:21:11.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a0406a350 con 0x7f9a040b5490 2024-04-25T11:21:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f99f401a070 con 0x7f9a040b5490 2024-04-25T11:21:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 0x7f99f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.387+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f99f408ea70 con 0x7f9a040b5490 2024-04-25T11:21:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.391+0000 7f9a027fc640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 0x7f99f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.391+0000 7f9a027fc640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 0x7f99f0071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f99f80b2000 tx=0x7f99f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.395+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f99f405b5f0 con 0x7f9a040b5490 2024-04-25T11:21:11.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.575+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a040cd9c0 con 0x7f9a040b5490 2024-04-25T11:21:11.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f99ebfff640 1 -- 172.21.15.136:0/2081901655 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6917 (secure 0 0 0) 0x7f99f405aef0 con 0x7f9a040b5490 2024-04-25T11:21:11.577 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:11.577 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:06.978801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:21:11.577 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 18 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 msgr2=0x7f99f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 0x7f99f0071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f99f80b2000 tx=0x7f99f8008040 comp rx=0 tx=0).stop 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 msgr2=0x7f9a041e5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c3f0 tx=0x7f99f400c8b0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 shutdown_connections 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f99f006f400 0x7f99f0071910 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a04156330 0x7f9a041e78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a040cdee0 0x7f9a041f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 --2- 172.21.15.136:0/2081901655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a040b5490 0x7f9a041e5400 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 >> 172.21.15.136:0/2081901655 conn(0x7f9a0406d6c0 msgr2=0x7f9a040c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 shutdown_connections 2024-04-25T11:21:11.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.579+0000 7f9a09805640 1 -- 172.21.15.136:0/2081901655 wait complete. 2024-04-25T11:21:11.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:11.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 Processor -- start 2024-04-25T11:21:11.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 -- start start 2024-04-25T11:21:11.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95800ca290 0x7f95800ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 0x7f958015a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958015d750 con 0x7f95800b5060 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f958015da30 con 0x7f95800ca290 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f9584eb3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f958015dd10 con 0x7f958015a430 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95800ca290 0x7f95800ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95800ca290 0x7f95800ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36076/0 (socket says 172.21.15.136:36076) 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 -- 172.21.15.136:0/2331161908 learned_addr learned my addr 172.21.15.136:0/2331161908 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957ed9d640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 0x7f958015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 -- 172.21.15.136:0/2331161908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 msgr2=0x7f958015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 0x7f958015a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 -- 172.21.15.136:0/2331161908 >> v1:172.21.15.136:6789/0 conn(0x7f95800b5060 legacy=0x7f95800c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957dd9b640 1 -- 172.21.15.136:0/2331161908 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958015dff0 con 0x7f95800ca290 2024-04-25T11:21:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957d59a640 1 -- 172.21.15.136:0/2331161908 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568373128 0 0) 0x7f958015d750 con 0x7f95800b5060 2024-04-25T11:21:11.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.747+0000 7f957ed9d640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 0x7f958015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:11.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f957dd9b640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800ca290 0x7f95800ca6a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9574009870 tx=0x7f957401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e820ce98633781d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f957d59a640 1 -- 172.21.15.136:0/2331161908 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95740b5070 con 0x7f95800ca290 2024-04-25T11:21:11.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f957d59a640 1 -- 172.21.15.136:0/2331161908 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95740a8a50 con 0x7f95800ca290 2024-04-25T11:21:11.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f957d59a640 1 -- 172.21.15.136:0/2331161908 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95740a8e20 con 0x7f95800ca290 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 -- 172.21.15.136:0/2331161908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800ca290 msgr2=0x7f95800ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800ca290 0x7f95800ca6a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9574009870 tx=0x7f957401ed50 comp rx=0 tx=0).stop 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 -- 172.21.15.136:0/2331161908 shutdown_connections 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f958015a430 0x7f958015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 --2- 172.21.15.136:0/2331161908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800ca290 0x7f95800ca6a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 -- 172.21.15.136:0/2331161908 >> 172.21.15.136:0/2331161908 conn(0x7f9580070f70 msgr2=0x7f9580068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 -- 172.21.15.136:0/2331161908 shutdown_connections 2024-04-25T11:21:11.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.751+0000 7f9584eb3640 1 -- 172.21.15.136:0/2331161908 wait complete. 2024-04-25T11:21:11.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 Processor -- start 2024-04-25T11:21:11.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- start start 2024-04-25T11:21:11.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 0x7f958015c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 0x7f9580147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f958015ad80 con 0x7f95800b5060 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9580159f10 con 0x7f958015a430 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f958015d3f0 con 0x7f95800ca290 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36080/0 (socket says 172.21.15.136:36080) 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 -- 172.21.15.136:0/582977090 learned_addr learned my addr 172.21.15.136:0/582977090 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957ed9d640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 0x7f9580147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 -- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 msgr2=0x7f958015c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957dd9b640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 0x7f958015c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 0x7f958015c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 -- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 msgr2=0x7f9580147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 0x7f9580147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 -- 172.21.15.136:0/582977090 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958015cb30 con 0x7f95800b5060 2024-04-25T11:21:11.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957ed9d640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 0x7f9580147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957dd9b640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 0x7f958015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f957e59c640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f957800c7a0 tx=0x7f957800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578017070 con 0x7f95800b5060 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9578003db0 con 0x7f95800b5060 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95801f3e30 con 0x7f95800b5060 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578005270 con 0x7f95800b5060 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95801f43f0 con 0x7f95800b5060 2024-04-25T11:21:11.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.755+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958006a350 con 0x7f95800b5060 2024-04-25T11:21:11.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.759+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f957801a070 con 0x7f95800b5060 2024-04-25T11:21:11.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.759+0000 7f956effd640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 0x7f9560071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:11.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.759+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f957808ec00 con 0x7f95800b5060 2024-04-25T11:21:11.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.759+0000 7f957dd9b640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 0x7f9560071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:11.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.759+0000 7f957dd9b640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 0x7f9560071940 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95740b2000 tx=0x7f95740aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:11.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.763+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9578095080 con 0x7f95800b5060 2024-04-25T11:21:11.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.919+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9580150d40 con 0x7f95800b5060 2024-04-25T11:21:11.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f956effd640 1 -- 172.21.15.136:0/582977090 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7f957805b0a0 con 0x7f95800b5060 2024-04-25T11:21:11.921 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:11.921 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 msgr2=0x7f9560071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 0x7f9560071940 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95740b2000 tx=0x7f95740aa070 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 msgr2=0x7f958015b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f957800c7a0 tx=0x7f957800cc60 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 shutdown_connections 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f958015a430 0x7f9580147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f956006f430 0x7f9560071940 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95800ca290 0x7f958015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 --2- 172.21.15.136:0/582977090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95800b5060 0x7f958015b250 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 >> 172.21.15.136:0/582977090 conn(0x7f9580070f70 msgr2=0x7f958014a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 shutdown_connections 2024-04-25T11:21:11.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:11.923+0000 7f9584eb3640 1 -- 172.21.15.136:0/582977090 wait complete. 2024-04-25T11:21:11.942 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:21:11.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:21:12.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 Processor -- start 2024-04-25T11:21:12.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 -- start start 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc1563e0 0x7f72cc06fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72cc070230 0x7f72cc070640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72cc0c97f0 con 0x7f72cc155fd0 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f72cc0c8c50 con 0x7f72cc070230 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d45d2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc15cea0 con 0x7f72cc1563e0 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d236e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f72cc155fd0 0x7f72cc0c90e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40876/0 (socket says 172.21.15.136:40876) 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d236e640 1 -- 172.21.15.136:0/2945009535 learned_addr learned my addr 172.21.15.136:0/2945009535 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d2b6f640 1 --2- 172.21.15.136:0/2945009535 >> v2:172.21.15.136:3300/0 conn(0x7f72cc070230 0x7f72cc070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d1b6d640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc1563e0 0x7f72cc06fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d136c640 1 -- 172.21.15.136:0/2945009535 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773323689 0 0) 0x7f72cc0c97f0 con 0x7f72cc155fd0 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d136c640 1 -- 172.21.15.136:0/2945009535 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b8003530 con 0x7f72cc155fd0 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d2b6f640 1 -- 172.21.15.136:0/2945009535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc1563e0 msgr2=0x7f72cc06fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d2b6f640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc1563e0 0x7f72cc06fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d2b6f640 1 -- 172.21.15.136:0/2945009535 >> v1:172.21.15.136:6789/0 conn(0x7f72cc155fd0 legacy=0x7f72cc0c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.095+0000 7f72d2b6f640 1 -- 172.21.15.136:0/2945009535 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc15cbc0 con 0x7f72cc070230 2024-04-25T11:21:12.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d2b6f640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc070230 0x7f72cc070640 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009d90 tx=0x7f72c401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29c51bf3f07b652 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d136c640 1 -- 172.21.15.136:0/2945009535 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c40b4070 con 0x7f72cc070230 2024-04-25T11:21:12.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d136c640 1 -- 172.21.15.136:0/2945009535 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c40af600 con 0x7f72cc070230 2024-04-25T11:21:12.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d136c640 1 -- 172.21.15.136:0/2945009535 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c40af9d0 con 0x7f72cc070230 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- 172.21.15.136:0/2945009535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc070230 msgr2=0x7f72cc070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc070230 0x7f72cc070640 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009d90 tx=0x7f72c401f150 comp rx=0 tx=0).stop 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- 172.21.15.136:0/2945009535 shutdown_connections 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc070230 0x7f72cc070640 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 --2- 172.21.15.136:0/2945009535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc1563e0 0x7f72cc06fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- 172.21.15.136:0/2945009535 >> 172.21.15.136:0/2945009535 conn(0x7f72cc0c9d50 msgr2=0x7f72cc0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- 172.21.15.136:0/2945009535 shutdown_connections 2024-04-25T11:21:12.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- 172.21.15.136:0/2945009535 wait complete. 2024-04-25T11:21:12.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 Processor -- start 2024-04-25T11:21:12.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.099+0000 7f72d45d2640 1 -- start start 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 0x7f72cc1489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72cc1563e0 0x7f72cc0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0ca8e0 con 0x7f72cc155fd0 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0c88e0 con 0x7f72cc1563e0 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc0caa60 con 0x7f72cc070230 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36108/0 (socket says 172.21.15.136:36108) 2024-04-25T11:21:12.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 -- 172.21.15.136:0/3255897398 learned_addr learned my addr 172.21.15.136:0/3255897398 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d236e640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 0x7f72cc1489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d2b6f640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72cc1563e0 0x7f72cc0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 -- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 msgr2=0x7f72cc1489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 0x7f72cc1489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 -- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72cc1563e0 msgr2=0x7f72cc0cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72cc1563e0 0x7f72cc0cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 -- 172.21.15.136:0/3255897398 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc0cc1a0 con 0x7f72cc155fd0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d236e640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 0x7f72cc1489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d1b6d640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00d800 tx=0x7f72bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc00b6c0 con 0x7f72cc155fd0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc019070 con 0x7f72cc155fd0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc0c7180 con 0x7f72cc155fd0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc00ba80 con 0x7f72cc155fd0 2024-04-25T11:21:12.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc0c7440 con 0x7f72cc155fd0 2024-04-25T11:21:12.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc06a350 con 0x7f72cc155fd0 2024-04-25T11:21:12.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f72bc00bc20 con 0x7f72cc155fd0 2024-04-25T11:21:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 0x7f72b0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.103+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f72bc08f970 con 0x7f72cc155fd0 2024-04-25T11:21:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.107+0000 7f72d236e640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 0x7f72b0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.107+0000 7f72d236e640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 0x7f72b0071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72c8005890 tx=0x7f72c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.111+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f72bc05c580 con 0x7f72cc155fd0 2024-04-25T11:21:12.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72cc070b00 con 0x7f72cc155fd0 2024-04-25T11:21:12.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72c2ffd640 1 -- 172.21.15.136:0/3255897398 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f72bc05be80 con 0x7f72cc155fd0 2024-04-25T11:21:12.269 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:21:12.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 msgr2=0x7f72b0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 0x7f72b0071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72c8005890 tx=0x7f72c8009040 comp rx=0 tx=0).stop 2024-04-25T11:21:12.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 msgr2=0x7f72cc1467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00d800 tx=0x7f72bc00dcc0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 shutdown_connections 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72cc1563e0 0x7f72cc0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72cc155fd0 0x7f72cc1467a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72b006f430 0x7f72b0071940 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 --2- 172.21.15.136:0/3255897398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72cc070230 0x7f72cc1489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 >> 172.21.15.136:0/3255897398 conn(0x7f72cc0c9d50 msgr2=0x7f72cc06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 shutdown_connections 2024-04-25T11:21:12.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.271+0000 7f72d45d2640 1 -- 172.21.15.136:0/3255897398 wait complete. 2024-04-25T11:21:12.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:21:12.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.435+0000 7f55a8121640 1 Processor -- start 2024-04-25T11:21:12.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.435+0000 7f55a8121640 1 -- start start 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a8121640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f55a00736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a8121640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a00c9a40 con 0x7f55a0073be0 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a8121640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a00c9480 con 0x7f55a00ca870 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a8121640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55a00c7390 con 0x7f55a0152520 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f55a0073be0 0x7f55a0162cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40890/0 (socket says 172.21.15.136:40890) 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 -- 172.21.15.136:0/900338059 learned_addr learned my addr 172.21.15.136:0/900338059 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a56bc640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f55a00736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174311294 0 0) 0x7f55a00c9a40 con 0x7f55a0073be0 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f558c003530 con 0x7f55a0073be0 2024-04-25T11:21:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2957344173 0 0) 0x7f558c003530 con 0x7f55a0073be0 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 msgr2=0x7f55a00736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f55a00736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 >> v1:172.21.15.157:6789/0 conn(0x7f55a00ca870 legacy=0x7f55a00c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a00c9760 con 0x7f55a0073be0 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f5598001a00 con 0x7f55a0073be0 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 >> v1:172.21.15.136:6789/0 conn(0x7f55a0073be0 legacy=0x7f55a0162cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f558c0039f0 0x7f558c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f558c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f558c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f55a00c7390 con 0x7f558c004d50 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f558c00bb30 con 0x7f558c0039f0 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f558c00ce40 con 0x7f55a0152520 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f558c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a5ebd640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f558c0039f0 0x7f558c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a56bc640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f558c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 -- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 msgr2=0x7f558c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0152520 0x7f558c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 -- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f558c0039f0 msgr2=0x7f558c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f558c0039f0 0x7f558c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a66be640 1 -- 172.21.15.136:0/900338059 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c003530 con 0x7f558c004d50 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.439+0000 7f55a5ebd640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f558c0039f0 0x7f558c003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a66be640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f558c005160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f559800a830 tx=0x7f559801d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a83ea97ab69734 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559800a480 con 0x7f558c004d50 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55980ad630 con 0x7f558c004d50 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a4ebb640 1 -- 172.21.15.136:0/900338059 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55980ada60 con 0x7f558c004d50 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- 172.21.15.136:0/900338059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 msgr2=0x7f558c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 --2- 172.21.15.136:0/900338059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f558c005160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f559800a830 tx=0x7f559801d360 comp rx=0 tx=0).stop 2024-04-25T11:21:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a66be640 1 -- 172.21.15.136:0/900338059 reap_dead start 2024-04-25T11:21:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- 172.21.15.136:0/900338059 shutdown_connections 2024-04-25T11:21:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- 172.21.15.136:0/900338059 >> 172.21.15.136:0/900338059 conn(0x7f55a00c9d50 msgr2=0x7f55a00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- 172.21.15.136:0/900338059 shutdown_connections 2024-04-25T11:21:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- 172.21.15.136:0/900338059 wait complete. 2024-04-25T11:21:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 Processor -- start 2024-04-25T11:21:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.443+0000 7f55a8121640 1 -- start start 2024-04-25T11:21:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55a01ee660 0x7f55a0151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0147810 0x7f55a0147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f55a00c8ad0 con 0x7f558c004d50 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0152930 con 0x7f55a01ee660 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55a00c8c50 con 0x7f55a0147810 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36112/0 (socket says 172.21.15.136:36112) 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 -- 172.21.15.136:0/3482174591 learned_addr learned my addr 172.21.15.136:0/3482174591 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 -- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0147810 msgr2=0x7f55a0147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a66be640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0147810 0x7f55a0147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0147810 0x7f55a0147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 -- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55a01ee660 msgr2=0x7f55a0151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a56bc640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55a01ee660 0x7f55a0151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55a01ee660 0x7f55a0151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 -- 172.21.15.136:0/3482174591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a014c3e0 con 0x7f558c004d50 2024-04-25T11:21:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a5ebd640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f55900106c0 tx=0x7f5590010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5590010e90 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559000e8d0 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a014c6c0 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559000eca0 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a015b8d0 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a006a350 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.447+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5590003600 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.451+0000 7f55967fc640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 0x7f5584071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.451+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5590091fd0 con 0x7f558c004d50 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.451+0000 7f55a56bc640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 0x7f5584071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.451+0000 7f55a56bc640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 0x7f5584071ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f55a00c82e0 tx=0x7f559c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.451+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5590028020 con 0x7f558c004d50 2024-04-25T11:21:12.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55a00759d0 con 0x7f558c004d50 2024-04-25T11:21:12.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55967fc640 1 -- 172.21.15.136:0/3482174591 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f5590016030 con 0x7f558c004d50 2024-04-25T11:21:12.601 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:12.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 msgr2=0x7f5584071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 0x7f5584071ab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f55a00c82e0 tx=0x7f559c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 msgr2=0x7f55a01f1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f55900106c0 tx=0x7f5590010b80 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 shutdown_connections 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f558406f5a0 0x7f5584071ab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55a0147810 0x7f55a0147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55a01ee660 0x7f55a0151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 --2- 172.21.15.136:0/3482174591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f558c004d50 0x7f55a01f1d70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 >> 172.21.15.136:0/3482174591 conn(0x7f55a00c9d50 msgr2=0x7f55a006e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 shutdown_connections 2024-04-25T11:21:12.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.603+0000 7f55a8121640 1 -- 172.21.15.136:0/3482174591 wait complete. 2024-04-25T11:21:12.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:21:12.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 Processor -- start 2024-04-25T11:21:12.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 -- start start 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 0x7f9f200c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9f200c8190 0x7f9f20073590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f2015d590 con 0x7f9f200729c0 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9f2015d340 con 0x7f9f200c8190 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f248de640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9f2015e1a0 con 0x7f9f20072dd0 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1ed9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9f200c8190 0x7f9f20073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9f200729c0 0x7f9f20142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40904/0 (socket says 172.21.15.136:40904) 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1e59c640 1 -- 172.21.15.136:0/2776618993 learned_addr learned my addr 172.21.15.136:0/2776618993 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f17fff640 1 --2- 172.21.15.136:0/2776618993 >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 0x7f9f200c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1ed9d640 1 -- 172.21.15.136:0/2776618993 >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 msgr2=0x7f9f200c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1dd9b640 1 -- 172.21.15.136:0/2776618993 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2586411480 0 0) 0x7f9f2015d590 con 0x7f9f200729c0 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1ed9d640 1 --2- 172.21.15.136:0/2776618993 >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 0x7f9f200c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1ed9d640 1 -- 172.21.15.136:0/2776618993 >> v1:172.21.15.136:6789/0 conn(0x7f9f200729c0 legacy=0x7f9f20142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f1ed9d640 1 -- 172.21.15.136:0/2776618993 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2015d870 con 0x7f9f200c8190 2024-04-25T11:21:12.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.779+0000 7f9f17fff640 1 --2- 172.21.15.136:0/2776618993 >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 0x7f9f200c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:12.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f1ed9d640 1 --2- 172.21.15.136:0/2776618993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f200c8190 0x7f9f20073590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9f20071fd0 tx=0x7f9f1801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d1df1201d16694 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f1dd9b640 1 -- 172.21.15.136:0/2776618993 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f180b2070 con 0x7f9f200c8190 2024-04-25T11:21:12.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f1dd9b640 1 -- 172.21.15.136:0/2776618993 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f180a57f0 con 0x7f9f200c8190 2024-04-25T11:21:12.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f1dd9b640 1 -- 172.21.15.136:0/2776618993 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f180a5bc0 con 0x7f9f200c8190 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- 172.21.15.136:0/2776618993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f200c8190 msgr2=0x7f9f20073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 --2- 172.21.15.136:0/2776618993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f200c8190 0x7f9f20073590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9f20071fd0 tx=0x7f9f1801c930 comp rx=0 tx=0).stop 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- 172.21.15.136:0/2776618993 shutdown_connections 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 --2- 172.21.15.136:0/2776618993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f200c8190 0x7f9f20073590 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 --2- 172.21.15.136:0/2776618993 >> v2:172.21.15.157:3300/0 conn(0x7f9f20072dd0 0x7f9f200c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- 172.21.15.136:0/2776618993 >> 172.21.15.136:0/2776618993 conn(0x7f9f200c9d50 msgr2=0x7f9f200ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- 172.21.15.136:0/2776618993 shutdown_connections 2024-04-25T11:21:12.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- 172.21.15.136:0/2776618993 wait complete. 2024-04-25T11:21:12.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 Processor -- start 2024-04-25T11:21:12.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.783+0000 7f9f248de640 1 -- start start 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 0x7f9f20164050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f200c8190 0x7f9f20163130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f200c76a0 con 0x7f9f20072dd0 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f200c7520 con 0x7f9f200729c0 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f20166ef0 con 0x7f9f200c8190 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36134/0 (socket says 172.21.15.136:36134) 2024-04-25T11:21:12.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 -- 172.21.15.136:0/1661660862 learned_addr learned my addr 172.21.15.136:0/1661660862 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f1ed9d640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f200c8190 0x7f9f20163130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f1e59c640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 0x7f9f20164050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 -- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f200c8190 msgr2=0x7f9f20163130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f200c8190 0x7f9f20163130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 -- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 msgr2=0x7f9f20164050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 0x7f9f20164050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 -- 172.21.15.136:0/1661660862 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f201f3170 con 0x7f9f20072dd0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f1e59c640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 0x7f9f20164050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f17fff640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c7f0 tx=0x7f9f0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08015070 con 0x7f9f20072dd0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f08004220 con 0x7f9f20072dd0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f20153880 con 0x7f9f20072dd0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08003620 con 0x7f9f20072dd0 2024-04-25T11:21:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f201f33f0 con 0x7f9f20072dd0 2024-04-25T11:21:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f2006a350 con 0x7f9f20072dd0 2024-04-25T11:21:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.787+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9f080039d0 con 0x7f9f20072dd0 2024-04-25T11:21:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.791+0000 7f9f16ffd640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 0x7f9f00071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.791+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9f0808f990 con 0x7f9f20072dd0 2024-04-25T11:21:12.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.791+0000 7f9f1e59c640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 0x7f9f00071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:12.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.791+0000 7f9f1e59c640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 0x7f9f00071910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f10007fd0 tx=0x7f9f10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:12.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.795+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f201f3170 con 0x7f9f20072dd0 2024-04-25T11:21:12.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.947+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9f20142ec0 con 0x7f9f20072dd0 2024-04-25T11:21:12.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f16ffd640 1 -- 172.21.15.136:0/1661660862 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9f0805c680 con 0x7f9f20072dd0 2024-04-25T11:21:12.948 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:21:12.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 msgr2=0x7f9f00071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 0x7f9f00071910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f10007fd0 tx=0x7f9f10009040 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 msgr2=0x7f9f201f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c7f0 tx=0x7f9f0800ccb0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 shutdown_connections 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f0006f400 0x7f9f00071910 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f200c8190 0x7f9f20163130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f20072dd0 0x7f9f201f3bf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 --2- 172.21.15.136:0/1661660862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f200729c0 0x7f9f20164050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 >> 172.21.15.136:0/1661660862 conn(0x7f9f200c9d50 msgr2=0x7f9f20141b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 shutdown_connections 2024-04-25T11:21:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:12.951+0000 7f9f248de640 1 -- 172.21.15.136:0/1661660862 wait complete. 2024-04-25T11:21:12.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:21:13.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 Processor -- start 2024-04-25T11:21:13.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 -- start start 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ae806c670 0x7f3ae806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ae80c9830 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ae80c8f90 con 0x7f3ae815df90 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aef899640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3ae80c8d60 con 0x7f3ae806c670 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aede36640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3ae80ce6c0 0x7f3ae8164740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40914/0 (socket says 172.21.15.136:40914) 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aede36640 1 -- 172.21.15.136:0/749785182 learned_addr learned my addr 172.21.15.136:0/749785182 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aece34640 1 --2- 172.21.15.136:0/749785182 >> v2:172.21.15.157:3300/0 conn(0x7f3ae806c670 0x7f3ae806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636007096 0 0) 0x7f3ae80c9830 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ad4003530 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3661241004 0 0) 0x7f3ad4003530 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 >> v2:172.21.15.157:3300/0 conn(0x7f3ae806c670 msgr2=0x7f3ae806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 --2- 172.21.15.136:0/749785182 >> v2:172.21.15.157:3300/0 conn(0x7f3ae806c670 0x7f3ae806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 >> v1:172.21.15.157:6789/0 conn(0x7f3ae815df90 legacy=0x7f3ae815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae80c9270 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f3ae0001a00 con 0x7f3ae80ce6c0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 >> v1:172.21.15.136:6789/0 conn(0x7f3ae80ce6c0 legacy=0x7f3ae8164740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ad40039c0 0x7f3ad4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad4004810 0x7f3ad4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 0x7f3ad400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad400c130 con 0x7f3ad40051d0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad400c2e0 con 0x7f3ad4004810 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad400c4e0 con 0x7f3ad40039c0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aede36640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 0x7f3ad400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aed635640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ad40039c0 0x7f3ad4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.123+0000 7f3aece34640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad4004810 0x7f3ad4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 -- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ad40039c0 msgr2=0x7f3ad4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ad40039c0 0x7f3ad4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 -- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad4004810 msgr2=0x7f3ad4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad4004810 0x7f3ad4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 -- 172.21.15.136:0/749785182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad4003530 con 0x7f3ad40051d0 2024-04-25T11:21:13.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aed635640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ad40039c0 0x7f3ad4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:13.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 -- 172.21.15.136:0/749785182 reap_dead start 2024-04-25T11:21:13.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aede36640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 0x7f3ad400b9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000a830 tx=0x7f3ae001d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895e51baf9196e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000a480 con 0x7f3ad40051d0 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae00ad630 con 0x7f3ad40051d0 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3adffff640 1 -- 172.21.15.136:0/749785182 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00ada20 con 0x7f3ad40051d0 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 -- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 msgr2=0x7f3ad400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 0x7f3ad400b9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000a830 tx=0x7f3ae001d2a0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 -- 172.21.15.136:0/749785182 shutdown_connections 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 --2- 172.21.15.136:0/749785182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ad40051d0 0x7f3ad400b9d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 -- 172.21.15.136:0/749785182 >> 172.21.15.136:0/749785182 conn(0x7f3ae806f9b0 msgr2=0x7f3ae806fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 -- 172.21.15.136:0/749785182 shutdown_connections 2024-04-25T11:21:13.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.127+0000 7f3aef899640 1 -- 172.21.15.136:0/749785182 wait complete. 2024-04-25T11:21:13.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 Processor -- start 2024-04-25T11:21:13.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- start start 2024-04-25T11:21:13.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad40051d0 0x7f3ae8155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ae815cc20 0x7f3ae8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae815a1f0 con 0x7f3ae80caec0 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae8159f10 con 0x7f3ad40051d0 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae815d3b0 con 0x7f3ae815cc20 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36144/0 (socket says 172.21.15.136:36144) 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 -- 172.21.15.136:0/3148465304 learned_addr learned my addr 172.21.15.136:0/3148465304 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aece34640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ae815cc20 0x7f3ae8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aed635640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad40051d0 0x7f3ae8155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 -- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ae815cc20 msgr2=0x7f3ae8156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ae815cc20 0x7f3ae8156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 -- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad40051d0 msgr2=0x7f3ae8155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad40051d0 0x7f3ae8155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 -- 172.21.15.136:0/3148465304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae81f0270 con 0x7f3ae80caec0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aede36640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000a830 tx=0x7f3ad4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00bc070 con 0x7f3ae80caec0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae00019d0 con 0x7f3ae80caec0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae81f0550 con 0x7f3ae80caec0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00ada00 con 0x7f3ae80caec0 2024-04-25T11:21:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.131+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae81f0b10 con 0x7f3ae80caec0 2024-04-25T11:21:13.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae806a350 con 0x7f3ae80caec0 2024-04-25T11:21:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3ae00bf070 con 0x7f3ae80caec0 2024-04-25T11:21:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3addffb640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 0x7f3acc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3ae01297e0 con 0x7f3ae80caec0 2024-04-25T11:21:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3aed635640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 0x7f3acc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.135+0000 7f3aed635640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 0x7f3acc071940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000c10 tx=0x7f3ae4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.139+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3ae00b4070 con 0x7f3ae80caec0 2024-04-25T11:21:13.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.287+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3ae81577a0 con 0x7f3ae80caec0 2024-04-25T11:21:13.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.287+0000 7f3addffb640 1 -- 172.21.15.136:0/3148465304 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f3ae0016fb0 con 0x7f3ae80caec0 2024-04-25T11:21:13.287 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 msgr2=0x7f3acc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 0x7f3acc071940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000c10 tx=0x7f3ae4009040 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 msgr2=0x7f3ae813dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000a830 tx=0x7f3ad4005840 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 shutdown_connections 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ae80caec0 0x7f3ae813dc10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3acc06f430 0x7f3acc071940 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ae815cc20 0x7f3ae8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 --2- 172.21.15.136:0/3148465304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ad40051d0 0x7f3ae8155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 >> 172.21.15.136:0/3148465304 conn(0x7f3ae806f9b0 msgr2=0x7f3ae8140dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 shutdown_connections 2024-04-25T11:21:13.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.291+0000 7f3aef899640 1 -- 172.21.15.136:0/3148465304 wait complete. 2024-04-25T11:21:13.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:21:13.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.443+0000 7fd9ebc98640 1 Processor -- start 2024-04-25T11:21:13.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.443+0000 7fd9ebc98640 1 -- start start 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9ebc98640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd9e40c4750 0x7fd9e4152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9ebc98640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e4158590 0x7fd9e41589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9ebc98640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e4071300 con 0x7fd9e40736a0 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9ebc98640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e40710d0 con 0x7fd9e40c4750 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9ebc98640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e40c95f0 con 0x7fd9e4158590 2024-04-25T11:21:13.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd9e40c4750 0x7fd9e4152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9233640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e4158590 0x7fd9e41589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd9e40c4750 0x7fd9e4152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:35854/0 (socket says 172.21.15.136:35854) 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 -- 172.21.15.136:0/129040510 learned_addr learned my addr 172.21.15.136:0/129040510 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 -- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e4158590 msgr2=0x7fd9e41589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 --2- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e4158590 0x7fd9e41589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 -- 172.21.15.136:0/129040510 >> v1:172.21.15.157:6789/0 conn(0x7fd9e40736a0 legacy=0x7fd9e415bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.447+0000 7fd9e9a34640 1 -- 172.21.15.136:0/129040510 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e40c7300 con 0x7fd9e40c4750 2024-04-25T11:21:13.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9e9a34640 1 --2- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e40c4750 0x7fd9e4152900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000bb10 tx=0x7fd9e001eb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57c02e85b55d6558 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9e8a32640 1 -- 172.21.15.136:0/129040510 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e00b4070 con 0x7fd9e40c4750 2024-04-25T11:21:13.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9e8a32640 1 -- 172.21.15.136:0/129040510 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e00a7960 con 0x7fd9e40c4750 2024-04-25T11:21:13.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9e8a32640 1 -- 172.21.15.136:0/129040510 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e00a7d30 con 0x7fd9e40c4750 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e40c4750 msgr2=0x7fd9e4152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e40c4750 0x7fd9e4152900 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd9e000bb10 tx=0x7fd9e001eb10 comp rx=0 tx=0).stop 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- 172.21.15.136:0/129040510 shutdown_connections 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e4158590 0x7fd9e41589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/129040510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e40c4750 0x7fd9e4152900 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- 172.21.15.136:0/129040510 >> 172.21.15.136:0/129040510 conn(0x7fd9e40c9d50 msgr2=0x7fd9e40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- 172.21.15.136:0/129040510 shutdown_connections 2024-04-25T11:21:13.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- 172.21.15.136:0/129040510 wait complete. 2024-04-25T11:21:13.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 Processor -- start 2024-04-25T11:21:13.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.451+0000 7fd9ebc98640 1 -- start start 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e40736a0 0x7fd9e41f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 0x7fd9e415abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e40ca870 con 0x7fd9e40c4750 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e40c9a80 con 0x7fd9e4158590 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e40ca9f0 con 0x7fd9e40736a0 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36150/0 (socket says 172.21.15.136:36150) 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 -- 172.21.15.136:0/1194617027 learned_addr learned my addr 172.21.15.136:0/1194617027 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9a34640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e40736a0 0x7fd9e41f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ea235640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 0x7fd9e415abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 -- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e40736a0 msgr2=0x7fd9e41f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e40736a0 0x7fd9e41f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 -- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 msgr2=0x7fd9e415abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 0x7fd9e415abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 -- 172.21.15.136:0/1194617027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e41442d0 con 0x7fd9e40c4750 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ea235640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 0x7fd9e415abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9e9233640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400e8d0 tx=0x7fd9d400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4017070 con 0x7fd9e40c4750 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d400c940 con 0x7fd9e40c4750 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e41f4990 con 0x7fd9e40c4750 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d400ccf0 con 0x7fd9e40c4750 2024-04-25T11:21:13.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e41f4f50 con 0x7fd9e40c4750 2024-04-25T11:21:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e406a350 con 0x7fd9e40c4750 2024-04-25T11:21:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd9d401a070 con 0x7fd9e40c4750 2024-04-25T11:21:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 0x7fd9c8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.455+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd9d408fc90 con 0x7fd9e40c4750 2024-04-25T11:21:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.459+0000 7fd9e9a34640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 0x7fd9c8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.459+0000 7fd9e9a34640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 0x7fd9c8071940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00b1000 tx=0x7fd9e00a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.459+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd9d405c210 con 0x7fd9e40c4750 2024-04-25T11:21:13.461 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:21:13.461+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:21:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.639+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9e414ca20 con 0x7fd9e40c4750 2024-04-25T11:21:13.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9da7fc640 1 -- 172.21.15.136:0/1194617027 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7fd9d4060510 con 0x7fd9e40c4750 2024-04-25T11:21:13.640 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:13.640 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:21:13.640 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 msgr2=0x7fd9c8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 0x7fd9c8071940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00b1000 tx=0x7fd9e00a9070 comp rx=0 tx=0).stop 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 msgr2=0x7fd9e41439a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400e8d0 tx=0x7fd9d400ecd0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 shutdown_connections 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd9e4158590 0x7fd9e415abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9e40c4750 0x7fd9e41439a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9c806f430 0x7fd9c8071940 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 --2- 172.21.15.136:0/1194617027 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd9e40736a0 0x7fd9e41f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 >> 172.21.15.136:0/1194617027 conn(0x7fd9e40c9d50 msgr2=0x7fd9e4157510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 shutdown_connections 2024-04-25T11:21:13.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.643+0000 7fd9ebc98640 1 -- 172.21.15.136:0/1194617027 wait complete. 2024-04-25T11:21:13.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:13.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.795+0000 7f3a5299b640 1 Processor -- start 2024-04-25T11:21:13.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.795+0000 7f3a5299b640 1 -- start start 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3a4c14f840 0x7f3a4c074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 0x7f3a4c073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c0c3c30 0x7f3a4c0c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4c06dc00 con 0x7f3a4c14f840 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4c14fc50 con 0x7f3a4c073200 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a5299b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c0c3750 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3a4c14f840 0x7f3a4c074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3a4c14f840 0x7f3a4c074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36154/0 (socket says 172.21.15.136:36154) 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 -- 172.21.15.136:0/1922340404 learned_addr learned my addr 172.21.15.136:0/1922340404 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:13.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a43fff640 1 --2- 172.21.15.136:0/1922340404 >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 0x7f3a4c073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a50f38640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c0c3c30 0x7f3a4c0c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 -- 172.21.15.136:0/1922340404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c0c3c30 msgr2=0x7f3a4c0c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c0c3c30 0x7f3a4c0c4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 -- 172.21.15.136:0/1922340404 >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 msgr2=0x7f3a4c073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/1922340404 >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 0x7f3a4c073610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a4bfff640 1 -- 172.21.15.136:0/1922340404 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c0c3190 con 0x7f3a4c14f840 2024-04-25T11:21:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.799+0000 7f3a43fff640 1 --2- 172.21.15.136:0/1922340404 >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 0x7f3a4c073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:13.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c14f840 0x7f3a4c074980 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0cb320 tx=0x7f3a3c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d875a7e8cacaed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a4b7fe640 1 -- 172.21.15.136:0/1922340404 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0b0070 con 0x7f3a4c14f840 2024-04-25T11:21:13.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a4b7fe640 1 -- 172.21.15.136:0/1922340404 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3c0ab600 con 0x7f3a4c14f840 2024-04-25T11:21:13.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a4b7fe640 1 -- 172.21.15.136:0/1922340404 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0ab9d0 con 0x7f3a4c14f840 2024-04-25T11:21:13.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- 172.21.15.136:0/1922340404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c14f840 msgr2=0x7f3a4c074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c14f840 0x7f3a4c074980 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0cb320 tx=0x7f3a3c01c520 comp rx=0 tx=0).stop 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- 172.21.15.136:0/1922340404 shutdown_connections 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c0c3c30 0x7f3a4c0c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 --2- 172.21.15.136:0/1922340404 >> v2:172.21.15.157:3300/0 conn(0x7f3a4c073200 0x7f3a4c073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 --2- 172.21.15.136:0/1922340404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c14f840 0x7f3a4c074980 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- 172.21.15.136:0/1922340404 >> 172.21.15.136:0/1922340404 conn(0x7f3a4c0ca5a0 msgr2=0x7f3a4c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- 172.21.15.136:0/1922340404 shutdown_connections 2024-04-25T11:21:13.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- 172.21.15.136:0/1922340404 wait complete. 2024-04-25T11:21:13.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 Processor -- start 2024-04-25T11:21:13.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.803+0000 7f3a5299b640 1 -- start start 2024-04-25T11:21:13.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c073200 0x7f3a4c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a4c14f840 0x7f3a4c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c153380 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c159c10 con 0x7f3a4c14f840 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c1597f0 con 0x7f3a4c073200 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36164/0 (socket says 172.21.15.136:36164) 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 -- 172.21.15.136:0/3815857521 learned_addr learned my addr 172.21.15.136:0/3815857521 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:13.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a50f38640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a4c14f840 0x7f3a4c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c073200 0x7f3a4c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 -- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c073200 msgr2=0x7f3a4c1f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c073200 0x7f3a4c1f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 -- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a4c14f840 msgr2=0x7f3a4c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a4c14f840 0x7f3a4c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 -- 172.21.15.136:0/3815857521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c0cd690 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a43fff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400d610 tx=0x7f3a3400dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34004030 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a340041d0 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c156c80 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34003620 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1f3470 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4c06a350 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.807+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3a340037c0 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.811+0000 7f3a497fa640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 0x7f3a2c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:13.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.811+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3a3408fc40 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.811+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 0x7f3a2c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:13.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.811+0000 7f3a4bfff640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 0x7f3a2c071940 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0ad000 tx=0x7f3a3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:13.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.815+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3a3405c850 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.975+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a4c1634d0 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a497fa640 1 -- 172.21.15.136:0/3815857521 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7f3a3405c150 con 0x7f3a4c0c3c30 2024-04-25T11:21:13.976 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:13.977 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 msgr2=0x7f3a2c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 0x7f3a2c071940 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0ad000 tx=0x7f3a3c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 msgr2=0x7f3a4c154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400d610 tx=0x7f3a3400dad0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 shutdown_connections 2024-04-25T11:21:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a4c14f840 0x7f3a4c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a4c0c3c30 0x7f3a4c154f00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a2c06f430 0x7f3a2c071940 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 --2- 172.21.15.136:0/3815857521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a4c073200 0x7f3a4c1f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 >> 172.21.15.136:0/3815857521 conn(0x7f3a4c0ca5a0 msgr2=0x7f3a4c14de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 shutdown_connections 2024-04-25T11:21:13.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:13.979+0000 7f3a5299b640 1 -- 172.21.15.136:0/3815857521 wait complete. 2024-04-25T11:21:13.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:21:14.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 Processor -- start 2024-04-25T11:21:14.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 -- start start 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8060070870 0x7f80600c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 0x7f80600c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8060143580 con 0x7f8060143170 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f806015dd20 con 0x7f8060070870 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f806668c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f806015e000 con 0x7f8060070c80 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8060070870 0x7f80600c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8060143170 0x7f806015d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:40920/0 (socket says 172.21.15.136:40920) 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805ffff640 1 -- 172.21.15.136:0/2418064680 learned_addr learned my addr 172.21.15.136:0/2418064680 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f8064c29640 1 --2- 172.21.15.136:0/2418064680 >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 0x7f80600c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805f7fe640 1 -- 172.21.15.136:0/2418064680 >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 msgr2=0x7f80600c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805f7fe640 1 --2- 172.21.15.136:0/2418064680 >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 0x7f80600c6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805f7fe640 1 -- 172.21.15.136:0/2418064680 >> v1:172.21.15.136:6789/0 conn(0x7f8060143170 legacy=0x7f806015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f805f7fe640 1 -- 172.21.15.136:0/2418064680 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806006fd90 con 0x7f8060070870 2024-04-25T11:21:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.139+0000 7f8064c29640 1 --2- 172.21.15.136:0/2418064680 >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 0x7f80600c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:14.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f805f7fe640 1 --2- 172.21.15.136:0/2418064680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80600c6960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8060075070 tx=0x7f805401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e473eb535cc1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:14.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f805effd640 1 -- 172.21.15.136:0/2418064680 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80540b5070 con 0x7f8060070870 2024-04-25T11:21:14.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f805effd640 1 -- 172.21.15.136:0/2418064680 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80540a8970 con 0x7f8060070870 2024-04-25T11:21:14.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f805effd640 1 -- 172.21.15.136:0/2418064680 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80540a8d40 con 0x7f8060070870 2024-04-25T11:21:14.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 -- 172.21.15.136:0/2418064680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 msgr2=0x7f80600c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 --2- 172.21.15.136:0/2418064680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80600c6960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8060075070 tx=0x7f805401ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:14.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 -- 172.21.15.136:0/2418064680 shutdown_connections 2024-04-25T11:21:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 --2- 172.21.15.136:0/2418064680 >> v2:172.21.15.157:3300/0 conn(0x7f8060070c80 0x7f80600c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 --2- 172.21.15.136:0/2418064680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80600c6960 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 -- 172.21.15.136:0/2418064680 >> 172.21.15.136:0/2418064680 conn(0x7f80600c9d50 msgr2=0x7f80600ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 -- 172.21.15.136:0/2418064680 shutdown_connections 2024-04-25T11:21:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.143+0000 7f806668c640 1 -- 172.21.15.136:0/2418064680 wait complete. 2024-04-25T11:21:14.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 Processor -- start 2024-04-25T11:21:14.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- start start 2024-04-25T11:21:14.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8060070c80 0x7f8060072da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 0x7f8060138c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f806015e2e0 con 0x7f8060070870 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8060070350 con 0x7f8060070c80 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80600712a0 con 0x7f8060143170 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36180/0 (socket says 172.21.15.136:36180) 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 -- 172.21.15.136:0/613643640 learned_addr learned my addr 172.21.15.136:0/613643640 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f8064c29640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 0x7f8060138c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805f7fe640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8060070c80 0x7f8060072da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 -- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 msgr2=0x7f8060138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 0x7f8060138c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 -- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8060070c80 msgr2=0x7f8060072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8060070c80 0x7f8060072da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 -- 172.21.15.136:0/613643640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80601f2e90 con 0x7f8060070870 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f8064c29640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 0x7f8060138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805ffff640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f805000c3a0 tx=0x7f805000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050017070 con 0x7f8060070870 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8050005cf0 con 0x7f8060070870 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8060159be0 con 0x7f8060070870 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500072f0 con 0x7f8060070870 2024-04-25T11:21:14.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806015a160 con 0x7f8060070870 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806006a350 con 0x7f8060070870 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.147+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f805001a070 con 0x7f8060070870 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.151+0000 7f805cff9640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 0x7f804c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.151+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f805008e9a0 con 0x7f8060070870 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.151+0000 7f805f7fe640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 0x7f804c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:14.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.151+0000 7f805f7fe640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 0x7f804c071940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80540b2000 tx=0x7f80540aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.155+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8060143580 con 0x7f8060070870 2024-04-25T11:21:14.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.335+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f806013e790 con 0x7f8060070870 2024-04-25T11:21:14.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.335+0000 7f805cff9640 1 -- 172.21.15.136:0/613643640 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f805005b520 con 0x7f8060070870 2024-04-25T11:21:14.335 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:14.336 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:21:14.336 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:21:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 msgr2=0x7f804c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 0x7f804c071940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80540b2000 tx=0x7f80540aa070 comp rx=0 tx=0).stop 2024-04-25T11:21:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 msgr2=0x7f80601f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f805000c3a0 tx=0x7f805000c860 comp rx=0 tx=0).stop 2024-04-25T11:21:14.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 shutdown_connections 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f804c06f430 0x7f804c071940 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8060143170 0x7f8060138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8060070c80 0x7f8060072da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 --2- 172.21.15.136:0/613643640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8060070870 0x7f80601f2950 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 >> 172.21.15.136:0/613643640 conn(0x7f80600c9d50 msgr2=0x7f80601424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 shutdown_connections 2024-04-25T11:21:14.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:14.339+0000 7f806668c640 1 -- 172.21.15.136:0/613643640 wait complete. 2024-04-25T11:21:14.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T11:20:19.386747+0000', 'modified': '2024-04-25T11:21:12.989866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4303, 'mds_1': 4297, 'mds_2': 4269, 'mds_3': 4260, 'mds_4': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.136:6835/430807169', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 430807169}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 430807169}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4260': {'gid': 4260, 'name': 'e', 'rank': 3, 'incarnation': 16, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.136:6837/1368493675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 1368493675}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 1368493675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4269': {'gid': 4269, 'name': 'b', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.157:6837/281980057', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6835', 'nonce': 281980057}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 281980057}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4297': {'gid': 4297, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.157:6836/694883637', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 694883637}, {'type': 'v1', 'addr': '172.21.15.157:6836', 'nonce': 694883637}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4303': {'gid': 4303, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.157:6839/1671830331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 1671830331}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 1671830331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4303, 'qdb_cluster': [4257, 4260, 4269, 4297, 4303]} 2024-04-25T11:21:14.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T11:21:14.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T11:21:15.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 Processor -- start 2024-04-25T11:21:15.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 -- start start 2024-04-25T11:21:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4060070870 0x7f40600c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4060070c80 0x7f40600c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4060143580 con 0x7f4060143170 2024-04-25T11:21:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f406015dd20 con 0x7f4060070c80 2024-04-25T11:21:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f40664ac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f406015e000 con 0x7f4060070870 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4060070c80 0x7f40600c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4060070c80 0x7f40600c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37136/0 (socket says 172.21.15.136:37136) 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 -- 172.21.15.136:0/1857970832 learned_addr learned my addr 172.21.15.136:0/1857970832 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f405f7fe640 1 --2- 172.21.15.136:0/1857970832 >> v2:172.21.15.157:3300/0 conn(0x7f4060070870 0x7f40600c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 -- 172.21.15.136:0/1857970832 >> v2:172.21.15.157:3300/0 conn(0x7f4060070870 msgr2=0x7f40600c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 --2- 172.21.15.136:0/1857970832 >> v2:172.21.15.157:3300/0 conn(0x7f4060070870 0x7f40600c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 -- 172.21.15.136:0/1857970832 >> v1:172.21.15.157:6789/0 conn(0x7f4060143170 legacy=0x7f406015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:15.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.123+0000 7f4064a49640 1 -- 172.21.15.136:0/1857970832 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406006fd90 con 0x7f4060070c80 2024-04-25T11:21:15.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f4064a49640 1 --2- 172.21.15.136:0/1857970832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c6ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4060157a80 tx=0x7f405801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39528e73d0c347a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:15.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f405effd640 1 -- 172.21.15.136:0/1857970832 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40580b4070 con 0x7f4060070c80 2024-04-25T11:21:15.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f405effd640 1 -- 172.21.15.136:0/1857970832 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40580af400 con 0x7f4060070c80 2024-04-25T11:21:15.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f405effd640 1 -- 172.21.15.136:0/1857970832 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40580af7d0 con 0x7f4060070c80 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 -- 172.21.15.136:0/1857970832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 msgr2=0x7f40600c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 --2- 172.21.15.136:0/1857970832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c6ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4060157a80 tx=0x7f405801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 -- 172.21.15.136:0/1857970832 shutdown_connections 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 --2- 172.21.15.136:0/1857970832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c6ea0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 --2- 172.21.15.136:0/1857970832 >> v2:172.21.15.157:3300/0 conn(0x7f4060070870 0x7f40600c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 -- 172.21.15.136:0/1857970832 >> 172.21.15.136:0/1857970832 conn(0x7f40600c9d50 msgr2=0x7f40600ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 -- 172.21.15.136:0/1857970832 shutdown_connections 2024-04-25T11:21:15.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.127+0000 7f40664ac640 1 -- 172.21.15.136:0/1857970832 wait complete. 2024-04-25T11:21:15.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 Processor -- start 2024-04-25T11:21:15.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- start start 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 0x7f40601f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 0x7f40600c3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f406015e2e0 con 0x7f4060070c80 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4060070350 con 0x7f4060070870 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40600712a0 con 0x7f4060143170 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 0x7f40601f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37150/0 (socket says 172.21.15.136:37150) 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 -- 172.21.15.136:0/830493409 learned_addr learned my addr 172.21.15.136:0/830493409 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 0x7f40601f2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:56834/0 (socket says 172.21.15.136:56834) 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f4064a49640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 0x7f40600c3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 -- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 msgr2=0x7f40600c3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 0x7f40600c3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 -- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 msgr2=0x7f40601f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 0x7f40601f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 -- 172.21.15.136:0/830493409 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40601f3530 con 0x7f4060070c80 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f4064a49640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 0x7f40600c3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405f7fe640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f405400e870 tx=0x7f405400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405cff9640 1 -- 172.21.15.136:0/830493409 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054017070 con 0x7f4060070c80 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405cff9640 1 -- 172.21.15.136:0/830493409 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405400c940 con 0x7f4060070c80 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40600ccf70 con 0x7f4060070c80 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f405cff9640 1 -- 172.21.15.136:0/830493409 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405400cd30 con 0x7f4060070c80 2024-04-25T11:21:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.131+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40601f5f90 con 0x7f4060070c80 2024-04-25T11:21:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405cff9640 1 -- 172.21.15.136:0/830493409 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f405401a070 con 0x7f4060070c80 2024-04-25T11:21:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405cff9640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 0x7f404c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405cff9640 1 -- 172.21.15.136:0/830493409 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f405408f870 con 0x7f4060070c80 2024-04-25T11:21:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 0x7f404c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 Processor -- start 2024-04-25T11:21:15.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 -- start start 2024-04-25T11:21:15.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 0x7f404c071910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4050001770 tx=0x7f4050009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef414a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x558ef3ffc160 con 0x558ef402ec20 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f91e20 con 0x558ef3c13280 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x558ef3f4aa20 con 0x558ef401aea0 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37152/0 (socket says 172.21.15.136:37152) 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 -- 172.21.15.136:0/988048518 learned_addr learned my addr 172.21.15.136:0/988048518 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f4064a49640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405f7fe640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef414a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 -- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 msgr2=0x558ef414a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef414a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 -- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 msgr2=0x558ef414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405ffff640 1 -- 172.21.15.136:0/988048518 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef3f4f100 con 0x558ef402ec20 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.135+0000 7f405f7fe640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef414a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f4064a49640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f405ffff640 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f405000fb40 tx=0x7f4050000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc596174aeedf51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f4045ffb640 1 -- 172.21.15.136:0/988048518 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050025070 con 0x558ef402ec20 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f4045ffb640 1 -- 172.21.15.136:0/988048518 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4050015bf0 con 0x558ef402ec20 2024-04-25T11:21:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f4045ffb640 1 -- 172.21.15.136:0/988048518 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050020400 con 0x558ef402ec20 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- 172.21.15.136:0/988048518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 msgr2=0x558ef417f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f405000fb40 tx=0x7f4050000ee0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- 172.21.15.136:0/988048518 shutdown_connections 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef414a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- 172.21.15.136:0/988048518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef417f5d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- 172.21.15.136:0/988048518 >> 172.21.15.136:0/988048518 conn(0x558ef3c52ff0 msgr2=0x558ef4182030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- 172.21.15.136:0/988048518 shutdown_connections 2024-04-25T11:21:15.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- 172.21.15.136:0/988048518 wait complete. 2024-04-25T11:21:15.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- start start 2024-04-25T11:21:15.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 Processor -- start 2024-04-25T11:21:15.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef402ec20 0x558ef4189e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f9f220 con 0x558ef401aea0 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f35ea0 con 0x558ef3c13280 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x558ef3f8cb00 con 0x558ef402ec20 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f406d809000 10 client.0 _ll_register_callbacks cb 0x558ef3f9edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f405f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f405f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37156/0 (socket says 172.21.15.136:37156) 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f405f7fe640 1 -- 172.21.15.136:0/2216438849 learned_addr learned my addr 172.21.15.136:0/2216438849 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f405ffff640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.139+0000 7f4064a49640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef402ec20 0x558ef4189e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 -- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef402ec20 msgr2=0x558ef4189e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef402ec20 0x558ef4189e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 -- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 msgr2=0x558ef414a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 -- 172.21.15.136:0/2216438849 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef3f31500 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f405408d9a0 tx=0x7f405405c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8a000c49ec4ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054021030 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f405405a9e0 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4050015bf0 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f405405b480 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 10 client.4902 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x558ef3f17ea0 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4050015bf0 con 0x558ef401aea0 2024-04-25T11:21:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f405405abe0 con 0x558ef401aea0 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 10 client.4902 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 10 client.4902 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 20 client.4902 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c377d0 0x558ef418a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 4 client.4902 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 --> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ef3f2abe0 con 0x558ef3c377d0 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405ffff640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c377d0 0x558ef418a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f403effd640 10 client.4902 ms_handle_connect on v2:172.21.15.136:6834/430807169 2024-04-25T11:21:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:15.143+0000 7f405ffff640 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c377d0 0x558ef418a840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a7dbbfd278b0e3b8 server_cookie=8630fbdddb0d7bb7 in_seq=0 out_seq=0 2024-04-25T11:21:17.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.995+0000 7f403effd640 1 -- 172.21.15.136:0/2216438849 <== mds.4 v2:172.21.15.136:6834/430807169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558ef3f2abe0 con 0x558ef3c377d0 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.995+0000 7f403effd640 10 client.4902 handle_command_reply: tid=0 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.995+0000 7f406d809000 1 client.4902 shutdown 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 msgr2=0x558ef41898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f405408d9a0 tx=0x7f405405c330 comp rx=0 tx=0).stop 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 shutdown_connections 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c377d0 0x558ef418a840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef402ec20 0x558ef4189e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef401aea0 0x558ef41898f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 --2- 172.21.15.136:0/2216438849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef414a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:17.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f403effd640 0 client.4902 ms_handle_reset on v2:172.21.15.136:6834/430807169 2024-04-25T11:21:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 >> 172.21.15.136:0/2216438849 conn(0x558ef3c52ff0 msgr2=0x558ef41811e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 shutdown_connections 2024-04-25T11:21:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 1 -- 172.21.15.136:0/2216438849 wait complete. 2024-04-25T11:21:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:17.999+0000 7f406d809000 20 client.4902 trim_cache size 0 max 16384 2024-04-25T11:21:18.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 Processor -- start 2024-04-25T11:21:18.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 -- start start 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef3c13280 0x558ef4145080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 0x558ef41455c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f277d0 con 0x558ef402ec20 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f91130 con 0x558ef401aea0 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x558ef3f96450 con 0x558ef3c13280 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f4064a49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f4064a49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37166/0 (socket says 172.21.15.136:37166) 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f4064a49640 1 -- 172.21.15.136:0/2866882829 learned_addr learned my addr 172.21.15.136:0/2866882829 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 0x558ef41455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.023+0000 7f405ffff640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef3c13280 0x558ef4145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 -- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef3c13280 msgr2=0x558ef4145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef3c13280 0x558ef4145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 -- 172.21.15.136:0/2866882829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 msgr2=0x558ef4145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 -- 172.21.15.136:0/2866882829 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ef414a880 con 0x558ef401aea0 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f4064a49640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 0x558ef41455c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f405405ba10 tx=0x7f405405bc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d48528bdbf7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f40457fa640 1 -- 172.21.15.136:0/2866882829 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054021070 con 0x558ef401aea0 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f40457fa640 1 -- 172.21.15.136:0/2866882829 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405405bd10 con 0x558ef401aea0 2024-04-25T11:21:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f40457fa640 1 -- 172.21.15.136:0/2866882829 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054060010 con 0x558ef401aea0 2024-04-25T11:21:18.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 msgr2=0x558ef41455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 0x558ef41455c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f405405ba10 tx=0x7f405405bc10 comp rx=0 tx=0).stop 2024-04-25T11:21:18.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- 172.21.15.136:0/2866882829 shutdown_connections 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef4145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef401aea0 0x558ef41455c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 --2- 172.21.15.136:0/2866882829 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef3c13280 0x558ef4145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- 172.21.15.136:0/2866882829 >> 172.21.15.136:0/2866882829 conn(0x558ef3c52ff0 msgr2=0x558ef41952c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- 172.21.15.136:0/2866882829 shutdown_connections 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- 172.21.15.136:0/2866882829 wait complete. 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 -- start start 2024-04-25T11:21:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.027+0000 7f406d809000 1 Processor -- start 2024-04-25T11:21:18.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef41f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef41f9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f6ea80 con 0x558ef402ec20 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x558ef3f9e0c0 con 0x558ef3c13280 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x558ef3f9f220 con 0x558ef401aea0 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 10 client.0 _ll_register_callbacks cb 0x7f405400ced0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37170/0 (socket says 172.21.15.136:37170) 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 -- 172.21.15.136:0/2985635370 learned_addr learned my addr 172.21.15.136:0/2985635370 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f405ffff640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef41f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f405f7fe640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef41f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 -- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 msgr2=0x558ef41f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef41f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 -- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 msgr2=0x558ef41f8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef41f8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 -- 172.21.15.136:0/2985635370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ef3f4ac70 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f4064a49640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4058009c90 tx=0x7f4058004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c22333ec2535cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40580b4040 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f40580af910 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405405bd10 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f40580afb10 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 10 client.4908 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x558ef4014180 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405405bd10 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f40580a8ce0 con 0x558ef402ec20 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 10 client.4908 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 10 client.4908 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 20 client.4908 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c25830 0x558ef41fa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 4 client.4908 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 --> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] -- command(tid 0: {"prefix": "status"}) -- 0x558ef3f9fa30 con 0x558ef3c25830 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f405ffff640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c25830 0x558ef41fa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 10 client.4908 ms_handle_connect on v2:172.21.15.136:6834/430807169 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f405ffff640 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c25830 0x558ef41fa7b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a05da4ac46584f67 server_cookie=bbfcf9da4afdd6ae in_seq=0 out_seq=0 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 1 -- 172.21.15.136:0/2985635370 <== mds.4 v2:172.21.15.136:6834/430807169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x558ef3f9fa30 con 0x558ef3c25830 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 10 client.4908 handle_command_reply: tid=0 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 client.4908 shutdown 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 msgr2=0x558ef41f9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4058009c90 tx=0x7f4058004a50 comp rx=0 tx=0).stop 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 shutdown_connections 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:6834/430807169,v1:172.21.15.136:6835/430807169] conn(0x558ef3c25830 0x558ef41fa7b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558ef402ec20 0x558ef41f9ad0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x558ef401aea0 0x558ef41f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 --2- 172.21.15.136:0/2985635370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558ef3c13280 0x558ef41f8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f403effd640 0 client.4908 ms_handle_reset on v2:172.21.15.136:6834/430807169 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 >> 172.21.15.136:0/2985635370 conn(0x558ef3c52ff0 msgr2=0x558ef4194890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 shutdown_connections 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 1 -- 172.21.15.136:0/2985635370 wait complete. 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.031+0000 7f406d809000 20 client.4908 trim_cache size 0 max 16384 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 4, 2024-04-25T11:21:18.036 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4257, 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 11.05234525, 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 19, 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 26, 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 65.601954539000005 2024-04-25T11:21:18.037 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 msgr2=0x7f404c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 0x7f404c071910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4050001770 tx=0x7f4050009040 comp rx=0 tx=0).stop 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 msgr2=0x7f40600c35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f405400e870 tx=0x7f405400ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 shutdown_connections 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f404c06f400 0x7f404c071910 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4060143170 0x7f40600c3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4060070c80 0x7f40600c35f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 --2- 172.21.15.136:0/830493409 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4060070870 0x7f40601f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 >> 172.21.15.136:0/830493409 conn(0x7f40600c9d50 msgr2=0x7f40600c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 shutdown_connections 2024-04-25T11:21:18.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.035+0000 7f40664ac640 1 -- 172.21.15.136:0/830493409 wait complete. 2024-04-25T11:21:18.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-25T11:21:18.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.203+0000 7f35ddd69640 1 Processor -- start 2024-04-25T11:21:18.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.203+0000 7f35ddd69640 1 -- start start 2024-04-25T11:21:18.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.203+0000 7f35ddd69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d8153190 0x7f35d80c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.203+0000 7f35ddd69640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f35d81594b0 0x7f35d8163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.203+0000 7f35ddd69640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35d80c8090 con 0x7f35d80c9570 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f35d80c8370 con 0x7f35d81594b0 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35d80c6aa0 con 0x7f35d8153190 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f35d81594b0 0x7f35d8163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d6ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f35d80c9570 0x7f35d8158da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60638/0 (socket says 172.21.15.136:60638) 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d6ffd640 1 -- 172.21.15.136:0/2708029594 learned_addr learned my addr 172.21.15.136:0/2708029594 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d77fe640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d8153190 0x7f35d80c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 -- 172.21.15.136:0/2708029594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d8153190 msgr2=0x7f35d80c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d8153190 0x7f35d80c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 -- 172.21.15.136:0/2708029594 >> v1:172.21.15.136:6789/0 conn(0x7f35d80c9570 legacy=0x7f35d8158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 -- 172.21.15.136:0/2708029594 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d80c6d80 con 0x7f35d81594b0 2024-04-25T11:21:18.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d67fc640 1 -- 172.21.15.136:0/2708029594 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068574817 0 0) 0x7f35d80c8090 con 0x7f35d80c9570 2024-04-25T11:21:18.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d7fff640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d81594b0 0x7f35d8163660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009d90 tx=0x7f35d001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d08e93813ecca46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d67fc640 1 -- 172.21.15.136:0/2708029594 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00b4070 con 0x7f35d81594b0 2024-04-25T11:21:18.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d67fc640 1 -- 172.21.15.136:0/2708029594 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d00af600 con 0x7f35d81594b0 2024-04-25T11:21:18.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35d67fc640 1 -- 172.21.15.136:0/2708029594 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00af9d0 con 0x7f35d81594b0 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- 172.21.15.136:0/2708029594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d81594b0 msgr2=0x7f35d8163660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d81594b0 0x7f35d8163660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009d90 tx=0x7f35d001f150 comp rx=0 tx=0).stop 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- 172.21.15.136:0/2708029594 shutdown_connections 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d81594b0 0x7f35d8163660 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 --2- 172.21.15.136:0/2708029594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d8153190 0x7f35d80c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- 172.21.15.136:0/2708029594 >> 172.21.15.136:0/2708029594 conn(0x7f35d80c9d50 msgr2=0x7f35d80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- 172.21.15.136:0/2708029594 shutdown_connections 2024-04-25T11:21:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.207+0000 7f35ddd69640 1 -- 172.21.15.136:0/2708029594 wait complete. 2024-04-25T11:21:18.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 Processor -- start 2024-04-25T11:21:18.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 -- start start 2024-04-25T11:21:18.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 0x7f35d81efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d81594b0 0x7f35d8154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35d80ca870 con 0x7f35d8153190 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f35d80c7340 con 0x7f35d80c9570 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35ddd69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35d80ca9f0 con 0x7f35d81594b0 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37190/0 (socket says 172.21.15.136:37190) 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 -- 172.21.15.136:0/3337136315 learned_addr learned my addr 172.21.15.136:0/3337136315 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d7fff640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d81594b0 0x7f35d8154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 0x7f35d81efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 -- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d81594b0 msgr2=0x7f35d8154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d81594b0 0x7f35d8154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 -- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 msgr2=0x7f35d81efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 0x7f35d81efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 -- 172.21.15.136:0/3337136315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d81f02e0 con 0x7f35d8153190 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 0x7f35d81efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.211+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35c800c440 tx=0x7f35c800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 -- 172.21.15.136:0/3337136315 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8018070 con 0x7f35d8153190 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 -- 172.21.15.136:0/3337136315 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c80074e0 con 0x7f35d8153190 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d81e1d30 con 0x7f35d8153190 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 -- 172.21.15.136:0/3337136315 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8004a10 con 0x7f35d8153190 2024-04-25T11:21:18.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d81e2010 con 0x7f35d8153190 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 -- 172.21.15.136:0/3337136315 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f35c801b070 con 0x7f35d8153190 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 0x7f35c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35bffff640 1 -- 172.21.15.136:0/3337136315 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35c808e870 con 0x7f35d8153190 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 0x7f35c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35e50c6000 1 Processor -- start 2024-04-25T11:21:18.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35e50c6000 1 -- start start 2024-04-25T11:21:18.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.215+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 0x7f35c4071910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009a40 tx=0x7f35cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f1941550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f1941a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5588f17f3160 con 0x5588f1825c20 2024-04-25T11:21:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5588f1788e20 con 0x5588f140a280 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5588f1741a20 con 0x5588f1811ea0 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37192/0 (socket says 172.21.15.136:37192) 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 -- 172.21.15.136:0/2050980477 learned_addr learned my addr 172.21.15.136:0/2050980477 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f1941550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d7fff640 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f1941a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 -- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 msgr2=0x5588f1941550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f1941550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 -- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 msgr2=0x5588f1941a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f1941a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 -- 172.21.15.136:0/2050980477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f1746100 con 0x5588f1825c20 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35d77fe640 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35cc010b10 tx=0x7f35cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a2a158b994efa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35bd7fa640 1 -- 172.21.15.136:0/2050980477 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc01c050 con 0x5588f1825c20 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35bd7fa640 1 -- 172.21.15.136:0/2050980477 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc026070 con 0x5588f1825c20 2024-04-25T11:21:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35bd7fa640 1 -- 172.21.15.136:0/2050980477 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc00f040 con 0x5588f1825c20 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- 172.21.15.136:0/2050980477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 msgr2=0x5588f19765d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35cc010b10 tx=0x7f35cc013000 comp rx=0 tx=0).stop 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- 172.21.15.136:0/2050980477 shutdown_connections 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f1941a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f1941550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 --2- 172.21.15.136:0/2050980477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f19765d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- 172.21.15.136:0/2050980477 >> 172.21.15.136:0/2050980477 conn(0x5588f1449ff0 msgr2=0x5588f1979030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- 172.21.15.136:0/2050980477 shutdown_connections 2024-04-25T11:21:18.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- 172.21.15.136:0/2050980477 wait complete. 2024-04-25T11:21:18.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 -- start start 2024-04-25T11:21:18.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.219+0000 7f35e50c6000 1 Processor -- start 2024-04-25T11:21:18.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 0x5588f19808f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f1980e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5588f1796220 con 0x5588f140a280 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5588f172cea0 con 0x5588f1811ea0 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5588f1783b00 con 0x5588f1825c20 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 10 client.0 _ll_register_callbacks cb 0x5588f1795dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37196/0 (socket says 172.21.15.136:37196) 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 -- 172.21.15.136:0/3924847188 learned_addr learned my addr 172.21.15.136:0/3924847188 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d7fff640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f1980e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 0x5588f19808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 -- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 msgr2=0x5588f1980e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f1980e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 -- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 msgr2=0x5588f19808f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 0x5588f19808f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 -- 172.21.15.136:0/3924847188 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f1728500 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 0x5588f19808f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d77fe640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f35cc006e90 tx=0x7f35cc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85cda71eea2d8f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35cc021c00 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35cc026070 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35cc021da0 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f35cc01c020 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 10 client.4923 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5588f170eea0 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc026070 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f35cc004020 con 0x5588f140a280 2024-04-25T11:21:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 10 client.4923 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 10 client.4923 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 20 client.4923 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f142e7d0 0x5588f1981840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 4 client.4923 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 --> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f1721be0 con 0x5588f142e7d0 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f142e7d0 0x5588f1981840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 10 client.4923 ms_handle_connect on v2:172.21.15.136:6836/1368493675 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f142e7d0 0x5588f1981840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1ebd7bead92f741d server_cookie=78e1b7d7f25470ad in_seq=0 out_seq=0 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 1 -- 172.21.15.136:0/3924847188 <== mds.3 v2:172.21.15.136:6836/1368493675 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5588f1721be0 con 0x5588f142e7d0 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 10 client.4923 handle_command_reply: tid=0 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 client.4923 shutdown 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 msgr2=0x5588f1941510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f35cc006e90 tx=0x7f35cc0046d0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 shutdown_connections 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f142e7d0 0x5588f1981840 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f1980e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f1811ea0 0x5588f19808f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35b67fc640 0 client.4923 ms_handle_reset on v2:172.21.15.136:6836/1368493675 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 --2- 172.21.15.136:0/3924847188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f140a280 0x5588f1941510 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 >> 172.21.15.136:0/3924847188 conn(0x5588f1449ff0 msgr2=0x5588f19781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 shutdown_connections 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 1 -- 172.21.15.136:0/3924847188 wait complete. 2024-04-25T11:21:18.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.223+0000 7f35e50c6000 20 client.4923 trim_cache size 0 max 16384 2024-04-25T11:21:18.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 Processor -- start 2024-04-25T11:21:18.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 -- start start 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f193f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f198b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35d81fcde0 con 0x5588f1825c20 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f35d81fd4b0 con 0x5588f140a280 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35d814d1d0 con 0x5588f1811ea0 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35d7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35d7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37206/0 (socket says 172.21.15.136:37206) 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.243+0000 7f35d7fff640 1 -- 172.21.15.136:0/21834699 learned_addr learned my addr 172.21.15.136:0/21834699 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d77fe640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f193f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f198b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 -- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 msgr2=0x5588f198b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f198b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 -- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 msgr2=0x5588f193f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f193f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 -- 172.21.15.136:0/21834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f1941880 con 0x5588f1825c20 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d77fe640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f193f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009d90 tx=0x7f35d0004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbf6bace170724a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35bcff9640 1 -- 172.21.15.136:0/21834699 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00b4040 con 0x5588f1825c20 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35bcff9640 1 -- 172.21.15.136:0/21834699 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d00ba6e0 con 0x5588f1825c20 2024-04-25T11:21:18.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35bcff9640 1 -- 172.21.15.136:0/21834699 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00baad0 con 0x5588f1825c20 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- 172.21.15.136:0/21834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 msgr2=0x5588f198be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009d90 tx=0x7f35d0004ac0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- 172.21.15.136:0/21834699 shutdown_connections 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1825c20 0x5588f198be40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1811ea0 0x5588f198b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- 172.21.15.136:0/21834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f193f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- 172.21.15.136:0/21834699 >> 172.21.15.136:0/21834699 conn(0x5588f1449ff0 msgr2=0x5588f1996500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- 172.21.15.136:0/21834699 shutdown_connections 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- 172.21.15.136:0/21834699 wait complete. 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- start start 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 Processor -- start 2024-04-25T11:21:18.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f19f7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f19fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5588f1788130 con 0x5588f1811ea0 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5588f171e7d0 con 0x5588f140a280 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5588f178d450 con 0x5588f1825c20 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35e50c6000 10 client.0 _ll_register_callbacks cb 0x7f35cc0109e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37210/0 (socket says 172.21.15.136:37210) 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d6ffd640 1 -- 172.21.15.136:0/1293043007 learned_addr learned my addr 172.21.15.136:0/1293043007 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d7fff640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f19fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d77fe640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f19f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.247+0000 7f35d6ffd640 1 -- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 msgr2=0x5588f19fbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f19fbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d6ffd640 1 -- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 msgr2=0x5588f19f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f19f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d6ffd640 1 -- 172.21.15.136:0/1293043007 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f1741c70 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d77fe640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f19f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d6ffd640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f35c800ce00 tx=0x7f35c8007030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fcf737fba140069 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8089070 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35c8085dd0 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35d00ba6e0 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f35c80742a0 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 10 client.4929 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5588f180b180 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d00ba6e0 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f35c808c020 con 0x5588f1811ea0 2024-04-25T11:21:18.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 10 client.4929 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 10 client.4929 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 20 client.4929 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f141c830 0x5588f19fe460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 4 client.4929 mds_command: new command op to 4260 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 --> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] -- command(tid 0: {"prefix": "status"}) -- 0x5588f1796a30 con 0x5588f141c830 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d77fe640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f141c830 0x5588f19fe460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 10 client.4929 ms_handle_connect on v2:172.21.15.136:6836/1368493675 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35d77fe640 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f141c830 0x5588f19fe460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5bf34a071336a196 server_cookie=5763e6096198bb7c in_seq=0 out_seq=0 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 1 -- 172.21.15.136:0/1293043007 <== mds.3 v2:172.21.15.136:6836/1368493675 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5588f1796a30 con 0x5588f141c830 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 10 client.4929 handle_command_reply: tid=0 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 client.4929 shutdown 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 msgr2=0x5588f19f97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f35c800ce00 tx=0x7f35c8007030 comp rx=0 tx=0).stop 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 shutdown_connections 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:6836/1368493675,v1:172.21.15.136:6837/1368493675] conn(0x5588f141c830 0x5588f19fe460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5588f1825c20 0x5588f19fbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35b67fc640 0 client.4929 ms_handle_reset on v2:172.21.15.136:6836/1368493675 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5588f1811ea0 0x5588f19f97b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 --2- 172.21.15.136:0/1293043007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5588f140a280 0x5588f19f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 >> 172.21.15.136:0/1293043007 conn(0x5588f1449ff0 msgr2=0x5588f19950d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 shutdown_connections 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 1 -- 172.21.15.136:0/1293043007 wait complete. 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35e50c6000 20 client.4929 trim_cache size 0 max 16384 2024-04-25T11:21:18.255 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 3, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4260, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 17.304859960000002, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 19, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 26, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 65.818185081999999 2024-04-25T11:21:18.256 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 msgr2=0x7f35c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 0x7f35c4071910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009a40 tx=0x7f35cc008040 comp rx=0 tx=0).stop 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 msgr2=0x7f35d814b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35c800c440 tx=0x7f35c800c900 comp rx=0 tx=0).stop 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 shutdown_connections 2024-04-25T11:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f35c406f400 0x7f35c4071910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35d81594b0 0x7f35d8154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35d8153190 0x7f35d814b620 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 --2- 172.21.15.136:0/3337136315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35d80c9570 0x7f35d81efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 >> 172.21.15.136:0/3337136315 conn(0x7f35d80c9d50 msgr2=0x7f35d8151f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 shutdown_connections 2024-04-25T11:21:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.251+0000 7f35ddd69640 1 -- 172.21.15.136:0/3337136315 wait complete. 2024-04-25T11:21:18.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T11:21:18.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 Processor -- start 2024-04-25T11:21:18.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 -- start start 2024-04-25T11:21:18.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f717815c0d0 0x7f71780cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7178166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717815c6f0 con 0x7f717815ff60 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f71780c6310 con 0x7f717815c0d0 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717fd15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f717815c990 con 0x7f7178160370 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717d2b0640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f717815ff60 0x7f71780ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60642/0 (socket says 172.21.15.136:60642) 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717d2b0640 1 -- 172.21.15.136:0/1628335765 learned_addr learned my addr 172.21.15.136:0/1628335765 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7178166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717dab1640 1 --2- 172.21.15.136:0/1628335765 >> v2:172.21.15.157:3300/0 conn(0x7f717815c0d0 0x7f71780cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897378982 0 0) 0x7f717815c6f0 con 0x7f717815ff60 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.415+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7164003530 con 0x7f717815ff60 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3981107996 0 0) 0x7f7164003530 con 0x7f717815ff60 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 msgr2=0x7f7178166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7178166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 >> v2:172.21.15.157:3300/0 conn(0x7f717815c0d0 msgr2=0x7f71780cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 --2- 172.21.15.136:0/1628335765 >> v2:172.21.15.157:3300/0 conn(0x7f717815c0d0 0x7f71780cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717815cc30 con 0x7f717815ff60 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f7168001680 con 0x7f717815ff60 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 >> v1:172.21.15.136:6789/0 conn(0x7f717815ff60 legacy=0x7f71780ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71640039c0 0x7f7164003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7164004810 0x7f7164004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7164005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f71780c6310 con 0x7f71640039c0 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f717815c990 con 0x7f7164004810 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f716400ba50 con 0x7f7178160370 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71640039c0 0x7f7164003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7164004810 0x7f7164004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7164005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 -- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 msgr2=0x7f7164005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7164005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 -- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7164004810 msgr2=0x7f7164004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7164004810 0x7f7164004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 -- 172.21.15.136:0/1628335765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7164003530 con 0x7f71640039c0 2024-04-25T11:21:18.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7178160370 0x7f7164005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717dab1640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71640039c0 0x7f7164003e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7170009ae0 tx=0x7f717001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a2ccfa7bf4cc4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700b2070 con 0x7f71640039c0 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71700a56a0 con 0x7f71640039c0 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717caaf640 1 -- 172.21.15.136:0/1628335765 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700096b0 con 0x7f71640039c0 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717fd15640 1 -- 172.21.15.136:0/1628335765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71640039c0 msgr2=0x7f7164003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717fd15640 1 --2- 172.21.15.136:0/1628335765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71640039c0 0x7f7164003e30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7170009ae0 tx=0x7f717001cab0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717e2b2640 1 -- 172.21.15.136:0/1628335765 reap_dead start 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717fd15640 1 -- 172.21.15.136:0/1628335765 shutdown_connections 2024-04-25T11:21:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.419+0000 7f717fd15640 1 -- 172.21.15.136:0/1628335765 >> 172.21.15.136:0/1628335765 conn(0x7f717806f9d0 msgr2=0x7f717806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- 172.21.15.136:0/1628335765 shutdown_connections 2024-04-25T11:21:18.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- 172.21.15.136:0/1628335765 wait complete. 2024-04-25T11:21:18.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 Processor -- start 2024-04-25T11:21:18.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- start start 2024-04-25T11:21:18.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f717806e290 0x7f71781468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f717814a6a0 0x7f717806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f71780cbc60 con 0x7f717814a290 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f717815fa40 con 0x7f717806e290 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717fd15640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f717815cf10 con 0x7f717814a6a0 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37230/0 (socket says 172.21.15.136:37230) 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 -- 172.21.15.136:0/2239529881 learned_addr learned my addr 172.21.15.136:0/2239529881 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 -- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f717814a6a0 msgr2=0x7f717806ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f717814a6a0 0x7f717806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 -- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f717806e290 msgr2=0x7f71781468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717dab1640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f717806e290 0x7f71781468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f717806e290 0x7f71781468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 -- 172.21.15.136:0/2239529881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71780c6f90 con 0x7f717814a290 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.423+0000 7f717d2b0640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7168011790 tx=0x7f7168011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 -- 172.21.15.136:0/2239529881 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168010040 con 0x7f717814a290 2024-04-25T11:21:18.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 -- 172.21.15.136:0/2239529881 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716801a070 con 0x7f717814a290 2024-04-25T11:21:18.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7178145ab0 con 0x7f717814a290 2024-04-25T11:21:18.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 -- 172.21.15.136:0/2239529881 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716800fb80 con 0x7f717814a290 2024-04-25T11:21:18.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71780c8220 con 0x7f717814a290 2024-04-25T11:21:18.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 -- 172.21.15.136:0/2239529881 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f716801f460 con 0x7f717814a290 2024-04-25T11:21:18.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 0x7f715c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f716e7fc640 1 -- 172.21.15.136:0/2239529881 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f71680927c0 con 0x7f717814a290 2024-04-25T11:21:18.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f717dab1640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 0x7f715c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f7187072000 1 Processor -- start 2024-04-25T11:21:18.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f7187072000 1 -- start start 2024-04-25T11:21:18.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.427+0000 7f717dab1640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 0x7f715c071910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7170009ae0 tx=0x7f7170008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6da05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 0x560bf6d6b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560bf6c1d160 con 0x560bf6834280 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560bf6bb2e20 con 0x560bf6c3bea0 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x560bf6b6ba20 con 0x560bf6c4fc20 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37234/0 (socket says 172.21.15.136:37234) 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 -- 172.21.15.136:0/1094357979 learned_addr learned my addr 172.21.15.136:0/1094357979 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717dab1640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6da05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 0x560bf6d6b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 -- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 msgr2=0x560bf6da05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6da05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 -- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 msgr2=0x560bf6d6b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 0x560bf6d6b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 -- 172.21.15.136:0/1094357979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bf6b70100 con 0x560bf6834280 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717dab1640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6da05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 0x560bf6d6b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f71740073e0 tx=0x7f71740077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff4ceb1828c4bab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f715b7fe640 1 -- 172.21.15.136:0/1094357979 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174014760 con 0x560bf6834280 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f715b7fe640 1 -- 172.21.15.136:0/1094357979 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717401a070 con 0x560bf6834280 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f715b7fe640 1 -- 172.21.15.136:0/1094357979 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174014b40 con 0x560bf6834280 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- 172.21.15.136:0/1094357979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 msgr2=0x560bf6d6ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f71740073e0 tx=0x7f71740077e0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- 172.21.15.136:0/1094357979 shutdown_connections 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6ba90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c3bea0 0x560bf6d6b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 --2- 172.21.15.136:0/1094357979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6da05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- 172.21.15.136:0/1094357979 >> 172.21.15.136:0/1094357979 conn(0x560bf6873ff0 msgr2=0x560bf6da3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- 172.21.15.136:0/1094357979 shutdown_connections 2024-04-25T11:21:18.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- 172.21.15.136:0/1094357979 wait complete. 2024-04-25T11:21:18.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 -- start start 2024-04-25T11:21:18.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.431+0000 7f7187072000 1 Processor -- start 2024-04-25T11:21:18.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6d6ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c4fc20 0x560bf6daa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560bf6bc0220 con 0x560bf6834280 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560bf6b56ea0 con 0x560bf6c4fc20 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x560bf6badb00 con 0x560bf6c3bea0 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 10 client.0 _ll_register_callbacks cb 0x560bf6bbfdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37248/0 (socket says 172.21.15.136:37248) 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 -- 172.21.15.136:0/3309216042 learned_addr learned my addr 172.21.15.136:0/3309216042 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717d2b0640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6d6ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717e2b2640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c4fc20 0x560bf6daa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 -- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 msgr2=0x560bf6d6ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6d6ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 -- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c4fc20 msgr2=0x560bf6daa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c4fc20 0x560bf6daa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 -- 172.21.15.136:0/3309216042 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bf6b52500 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717dab1640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f71700b0040 tx=0x7f717001b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fa41ad3df487f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700b2040 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f71700ad8c0 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f717401a070 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f71700d0050 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 10 client.4941 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x560bf6b38ea0 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717401a070 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f71700d3020 con 0x560bf6834280 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 10 client.4941 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 10 client.4941 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 20 client.4941 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf68587d0 0x560bf6dab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 4 client.4941 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 --> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bf6b4bbe0 con 0x560bf68587d0 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717d2b0640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf68587d0 0x560bf6dab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f7158ff9640 10 client.4941 ms_handle_connect on v2:172.21.15.157:6835/281980057 2024-04-25T11:21:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.435+0000 7f717d2b0640 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf68587d0 0x560bf6dab690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe77d95162dd5d37 server_cookie=3d4dcc3e0801861a in_seq=0 out_seq=0 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7158ff9640 1 -- 172.21.15.136:0/3309216042 <== mds.2 v2:172.21.15.157:6835/281980057 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560bf6b4bbe0 con 0x560bf68587d0 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7158ff9640 10 client.4941 handle_command_reply: tid=0 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 client.4941 shutdown 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 msgr2=0x560bf6d6b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f71700b0040 tx=0x7f717001b2f0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 shutdown_connections 2024-04-25T11:21:18.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf68587d0 0x560bf6dab690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6c4fc20 0x560bf6daa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6d6ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 --2- 172.21.15.136:0/3309216042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6834280 0x560bf6d6b4e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7158ff9640 0 client.4941 ms_handle_reset on v2:172.21.15.157:6835/281980057 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 >> 172.21.15.136:0/3309216042 conn(0x560bf6873ff0 msgr2=0x560bf6da21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 shutdown_connections 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 1 -- 172.21.15.136:0/3309216042 wait complete. 2024-04-25T11:21:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.439+0000 7f7187072000 20 client.4941 trim_cache size 0 max 16384 2024-04-25T11:21:18.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 Processor -- start 2024-04-25T11:21:18.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- start start 2024-04-25T11:21:18.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6dc00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6dc0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7178200a80 con 0x560bf6c4fc20 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7178201150 con 0x560bf6834280 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f71781ff230 con 0x560bf6c3bea0 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37260/0 (socket says 172.21.15.136:37260) 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 -- 172.21.15.136:0/1156843986 learned_addr learned my addr 172.21.15.136:0/1156843986 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6dc0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717dab1640 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6dc00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 -- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 msgr2=0x560bf6dc0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6dc0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 -- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 msgr2=0x560bf6dc00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6dc00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 -- 172.21.15.136:0/1156843986 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bf6d63120 con 0x560bf6c4fc20 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7174017040 tx=0x7f7174012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9ce06f4c34bd64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f715affd640 1 -- 172.21.15.136:0/1156843986 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174022070 con 0x560bf6c4fc20 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f715affd640 1 -- 172.21.15.136:0/1156843986 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717401a050 con 0x560bf6c4fc20 2024-04-25T11:21:18.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f715affd640 1 -- 172.21.15.136:0/1156843986 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174003e00 con 0x560bf6c4fc20 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- 172.21.15.136:0/1156843986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 msgr2=0x560bf6da3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7174017040 tx=0x7f7174012000 comp rx=0 tx=0).stop 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- 172.21.15.136:0/1156843986 shutdown_connections 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c4fc20 0x560bf6da3f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c3bea0 0x560bf6dc0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 --2- 172.21.15.136:0/1156843986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6dc00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- 172.21.15.136:0/1156843986 >> 172.21.15.136:0/1156843986 conn(0x560bf6873ff0 msgr2=0x560bf6e14fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- 172.21.15.136:0/1156843986 shutdown_connections 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.459+0000 7f7187072000 1 -- 172.21.15.136:0/1156843986 wait complete. 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- start start 2024-04-25T11:21:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 Processor -- start 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6da9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6dddff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560bf6bb2130 con 0x560bf6c3bea0 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x560bf6b487d0 con 0x560bf6834280 2024-04-25T11:21:18.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x560bf6bb7450 con 0x560bf6c4fc20 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 10 client.0 _ll_register_callbacks cb 0x7f71700b0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37266/0 (socket says 172.21.15.136:37266) 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 -- 172.21.15.136:0/1173800830 learned_addr learned my addr 172.21.15.136:0/1173800830 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717e2b2640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6dddff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717dab1640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6da9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 -- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 msgr2=0x560bf6dddff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6dddff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 -- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 msgr2=0x560bf6da9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6da9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 -- 172.21.15.136:0/1173800830 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bf6b6bc70 con 0x560bf6c3bea0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717dab1640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6da9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717d2b0640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7168001ca0 tx=0x7f7168000ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3208fa190e96e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168062f60 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7168090370 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f717401a050 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f716807f050 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 10 client.4947 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x560bf6c35180 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717401a050 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f7168080cf0 con 0x560bf6c3bea0 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 10 client.4947 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 10 client.4947 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 20 client.4947 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf6846830 0x560bf6e24ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 4 client.4947 mds_command: new command op to 4269 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 --> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] -- command(tid 0: {"prefix": "status"}) -- 0x560bf6bc0a30 con 0x560bf6846830 2024-04-25T11:21:18.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717dab1640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf6846830 0x560bf6e24ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f7158ff9640 10 client.4947 ms_handle_connect on v2:172.21.15.157:6835/281980057 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.463+0000 7f717dab1640 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf6846830 0x560bf6e24ee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6bbc93d4a451ad9 server_cookie=b87c698cfdfbf0f in_seq=0 out_seq=0 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7158ff9640 1 -- 172.21.15.136:0/1173800830 <== mds.2 v2:172.21.15.157:6835/281980057 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x560bf6bc0a30 con 0x560bf6846830 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7158ff9640 10 client.4947 handle_command_reply: tid=0 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 client.4947 shutdown 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 msgr2=0x560bf6dddab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7168001ca0 tx=0x7f7168000ff0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 shutdown_connections 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:6835/281980057,v1:172.21.15.157:6837/281980057] conn(0x560bf6846830 0x560bf6e24ee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x560bf6c4fc20 0x560bf6dddff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560bf6c3bea0 0x560bf6dddab0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7158ff9640 0 client.4947 ms_handle_reset on v2:172.21.15.157:6835/281980057 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 --2- 172.21.15.136:0/1173800830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bf6834280 0x560bf6da9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 >> 172.21.15.136:0/1173800830 conn(0x560bf6873ff0 msgr2=0x560bf6dbfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 shutdown_connections 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 1 -- 172.21.15.136:0/1173800830 wait complete. 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f7187072000 20 client.4947 trim_cache size 0 max 16384 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 2, 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4269, 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 22.572495951000001, 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 19, 2024-04-25T11:21:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 26, 2024-04-25T11:21:18.470 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T11:21:18.470 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 66.018121020999999 2024-04-25T11:21:18.470 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:18.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 msgr2=0x7f715c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 0x7f715c071910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7170009ae0 tx=0x7f7170008040 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 msgr2=0x7f717806c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7168011790 tx=0x7f7168011c50 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 shutdown_connections 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f715c06f400 0x7f715c071910 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f717814a6a0 0x7f717806ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f717814a290 0x7f717806c770 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 --2- 172.21.15.136:0/2239529881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f717806e290 0x7f71781468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 >> 172.21.15.136:0/2239529881 conn(0x7f717806f9d0 msgr2=0x7f71781547d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 shutdown_connections 2024-04-25T11:21:18.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.467+0000 7f717fd15640 1 -- 172.21.15.136:0/2239529881 wait complete. 2024-04-25T11:21:18.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T11:21:18.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 Processor -- start 2024-04-25T11:21:18.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 -- start start 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90d40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90d40c9af0 con 0x7f90d40c96e0 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90d40cb220 con 0x7f90d415a3d0 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90db211640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90d406cc10 con 0x7f90d40c79e0 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d8fad640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f90d40c96e0 0x7f90d41540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60650/0 (socket says 172.21.15.136:60650) 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d8fad640 1 -- 172.21.15.136:0/45618586 learned_addr learned my addr 172.21.15.136:0/45618586 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d3fff640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90d40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895086749 0 0) 0x7f90d40c9af0 con 0x7f90d40c96e0 2024-04-25T11:21:18.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90bc003530 con 0x7f90d40c96e0 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4033423587 0 0) 0x7f90bc003530 con 0x7f90d40c96e0 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 msgr2=0x7f90d40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90d40c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 >> v1:172.21.15.157:6789/0 conn(0x7f90d415a3d0 legacy=0x7f90d415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d40cb500 con 0x7f90d40c96e0 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f90c4001680 con 0x7f90d40c96e0 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 >> v1:172.21.15.136:6789/0 conn(0x7f90d40c96e0 legacy=0x7f90d41540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc0039f0 0x7f90bc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc004810 0x7f90bc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90bc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f90d406cc10 con 0x7f90bc004810 2024-04-25T11:21:18.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f90bc00bb00 con 0x7f90bc0039f0 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90bc00bd00 con 0x7f90d40c79e0 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.631+0000 7f90d3fff640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc004810 0x7f90bc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d8fad640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc0039f0 0x7f90bc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d97ae640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90bc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 -- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 msgr2=0x7f90bc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d40c79e0 0x7f90bc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 -- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc0039f0 msgr2=0x7f90bc003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc0039f0 0x7f90bc003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 -- 172.21.15.136:0/45618586 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc003530 con 0x7f90bc004810 2024-04-25T11:21:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d3fff640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc004810 0x7f90bc004c20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178a78d229dafe26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80b2070 con 0x7f90bc004810 2024-04-25T11:21:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c80a5550 con 0x7f90bc004810 2024-04-25T11:21:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d37fe640 1 -- 172.21.15.136:0/45618586 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80096b0 con 0x7f90bc004810 2024-04-25T11:21:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 -- 172.21.15.136:0/45618586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc004810 msgr2=0x7f90bc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 --2- 172.21.15.136:0/45618586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc004810 0x7f90bc004c20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c801ca70 comp rx=0 tx=0).stop 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90d97ae640 1 -- 172.21.15.136:0/45618586 reap_dead start 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 -- 172.21.15.136:0/45618586 shutdown_connections 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 -- 172.21.15.136:0/45618586 >> 172.21.15.136:0/45618586 conn(0x7f90d406f9d0 msgr2=0x7f90d406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 -- 172.21.15.136:0/45618586 shutdown_connections 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.635+0000 7f90db211640 1 -- 172.21.15.136:0/45618586 wait complete. 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 Processor -- start 2024-04-25T11:21:18.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- start start 2024-04-25T11:21:18.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc004810 0x7f90d414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d4147fe0 0x7f90d41f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f90d40ced40 con 0x7f90d4154870 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f90d40cf0f0 con 0x7f90bc004810 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90d4155050 con 0x7f90d4147fe0 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37278/0 (socket says 172.21.15.136:37278) 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 -- 172.21.15.136:0/3901029916 learned_addr learned my addr 172.21.15.136:0/3901029916 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d3fff640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d4147fe0 0x7f90d41f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc004810 0x7f90d414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 -- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d4147fe0 msgr2=0x7f90d41f0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d4147fe0 0x7f90d41f0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 -- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc004810 msgr2=0x7f90d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc004810 0x7f90d414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 -- 172.21.15.136:0/3901029916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d4147df0 con 0x7f90d4154870 2024-04-25T11:21:18.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90d4075160 tx=0x7f90cc00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d17fa640 1 -- 172.21.15.136:0/3901029916 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc003980 con 0x7f90d4154870 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d17fa640 1 -- 172.21.15.136:0/3901029916 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90cc018070 con 0x7f90d4154870 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d414e170 con 0x7f90d4154870 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90d17fa640 1 -- 172.21.15.136:0/3901029916 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc003d60 con 0x7f90d4154870 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.639+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d41f36b0 con 0x7f90d4154870 2024-04-25T11:21:18.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90d17fa640 1 -- 172.21.15.136:0/3901029916 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f90cc0050d0 con 0x7f90d4154870 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90d17fa640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 0x7f90c0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90d17fa640 1 -- 172.21.15.136:0/3901029916 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f90cc08f370 con 0x7f90d4154870 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 0x7f90c0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 Processor -- start 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 -- start start 2024-04-25T11:21:18.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 0x7f90c0071940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90c4005340 tx=0x7f90c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8bbc20 0x55d8cea0c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8ce9d7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.643+0000 7f90e256e000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce889160 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce81ee20 con 0x55d8ce4a0280 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55d8ce7d7a20 con 0x55d8ce8bbc20 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37288/0 (socket says 172.21.15.136:37288) 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 -- 172.21.15.136:0/58956180 learned_addr learned my addr 172.21.15.136:0/58956180 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d8fad640 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8bbc20 0x55d8cea0c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d97ae640 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8ce9d7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 -- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8bbc20 msgr2=0x55d8cea0c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8bbc20 0x55d8cea0c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 -- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 msgr2=0x55d8ce9d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8ce9d7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 -- 172.21.15.136:0/58956180 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8ce7dc100 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90d3fff640 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db9d197edf8208d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90ba7fc640 1 -- 172.21.15.136:0/58956180 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80b2040 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90ba7fc640 1 -- 172.21.15.136:0/58956180 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c8004220 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90ba7fc640 1 -- 172.21.15.136:0/58956180 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c8003620 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- 172.21.15.136:0/58956180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 msgr2=0x55d8ce9d7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c8004740 comp rx=0 tx=0).stop 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- 172.21.15.136:0/58956180 shutdown_connections 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8ce9d7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8a7ea0 0x55d8ce9d7550 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 --2- 172.21.15.136:0/58956180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8bbc20 0x55d8cea0c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- 172.21.15.136:0/58956180 >> 172.21.15.136:0/58956180 conn(0x55d8ce4dfff0 msgr2=0x55d8cea0f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- 172.21.15.136:0/58956180 shutdown_connections 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- 172.21.15.136:0/58956180 wait complete. 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 -- start start 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.647+0000 7f90e256e000 1 Processor -- start 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea0b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea0bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce82c220 con 0x55d8ce8bbc20 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce7c2ea0 con 0x55d8ce4a0280 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55d8ce819b00 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 10 client.0 _ll_register_callbacks cb 0x55d8ce82bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37304/0 (socket says 172.21.15.136:37304) 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 -- 172.21.15.136:0/3906921745 learned_addr learned my addr 172.21.15.136:0/3906921745 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d3fff640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea0bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea0b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 -- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 msgr2=0x55d8cea0bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea0bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 -- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 msgr2=0x55d8cea0b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea0b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 -- 172.21.15.136:0/3906921745 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8ce7be500 con 0x55d8ce8bbc20 2024-04-25T11:21:18.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d97ae640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90cc057d50 tx=0x7f90cc05b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adee43d63398dd75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc070db0 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f90cc05a9c0 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90c8004220 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90cc05b760 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 10 client.4959 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55d8ce7a4ea0 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c8004220 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f90cc0570d0 con 0x55d8ce8bbc20 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 10 client.4959 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 10 client.4959 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 20 client.4959 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4c47d0 0x55d8ce9d4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 4 client.4959 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:21:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 --> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8ce7b7be0 con 0x55d8ce4c47d0 2024-04-25T11:21:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4c47d0 0x55d8ce9d4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90d8fad640 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4c47d0 0x55d8ce9d4380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d9bf125db7b8d9a server_cookie=ceb6d8a0d1b86611 in_seq=0 out_seq=0 2024-04-25T11:21:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.651+0000 7f90b37fe640 10 client.4959 ms_handle_connect on v2:172.21.15.157:6834/694883637 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90b37fe640 1 -- 172.21.15.136:0/3906921745 <== mds.1 v2:172.21.15.157:6834/694883637 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d8ce7b7be0 con 0x55d8ce4c47d0 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90b37fe640 10 client.4959 handle_command_reply: tid=0 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 client.4959 shutdown 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 msgr2=0x55d8ce9d3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f90cc057d50 tx=0x7f90cc05b170 comp rx=0 tx=0).stop 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 shutdown_connections 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4c47d0 0x55d8ce9d4380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce8bbc20 0x55d8ce9d3640 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea0bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90b37fe640 0 client.4959 ms_handle_reset on v2:172.21.15.157:6834/694883637 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 --2- 172.21.15.136:0/3906921745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea0b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 >> 172.21.15.136:0/3906921745 conn(0x55d8ce4dfff0 msgr2=0x55d8cea0c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 shutdown_connections 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 1 -- 172.21.15.136:0/3906921745 wait complete. 2024-04-25T11:21:18.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.655+0000 7f90e256e000 20 client.4959 trim_cache size 0 max 16384 2024-04-25T11:21:18.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.671+0000 7f90e256e000 1 Processor -- start 2024-04-25T11:21:18.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.671+0000 7f90e256e000 1 -- start start 2024-04-25T11:21:18.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce4a0280 0x55d8cea81ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce8a7ea0 0x55d8cea82020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f90d41feec0 con 0x55d8ce4dfff0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f90d41fcf70 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55d8ce7b47d0 con 0x55d8ce4a0280 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37306/0 (socket says 172.21.15.136:37306) 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 -- 172.21.15.136:0/1405937104 learned_addr learned my addr 172.21.15.136:0/1405937104 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 -- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce4a0280 msgr2=0x55d8cea81ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce4a0280 0x55d8cea81ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 -- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce8a7ea0 msgr2=0x55d8cea82020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce8a7ea0 0x55d8cea82020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 -- 172.21.15.136:0/1405937104 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8ce7d7c70 con 0x55d8ce4dfff0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55d8ce7dd3d0 tx=0x7f90c4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8487f3ff612a5832 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90b9ffb640 1 -- 172.21.15.136:0/1405937104 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4010040 con 0x55d8ce4dfff0 2024-04-25T11:21:18.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90b9ffb640 1 -- 172.21.15.136:0/1405937104 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c4031070 con 0x55d8ce4dfff0 2024-04-25T11:21:18.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90b9ffb640 1 -- 172.21.15.136:0/1405937104 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4013b20 con 0x55d8ce4dfff0 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- 172.21.15.136:0/1405937104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 msgr2=0x55d8cea815a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55d8ce7dd3d0 tx=0x7f90c4016000 comp rx=0 tx=0).stop 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- 172.21.15.136:0/1405937104 shutdown_connections 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce8a7ea0 0x55d8cea82020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce4a0280 0x55d8cea81ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 --2- 172.21.15.136:0/1405937104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea815a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- 172.21.15.136:0/1405937104 >> 172.21.15.136:0/1405937104 conn(0x55d8cea0afa0 msgr2=0x55d8cea2caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- 172.21.15.136:0/1405937104 shutdown_connections 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- 172.21.15.136:0/1405937104 wait complete. 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 -- start start 2024-04-25T11:21:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.675+0000 7f90e256e000 1 Processor -- start 2024-04-25T11:21:18.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea2c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea2a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce823450 con 0x55d8ce4dfff0 2024-04-25T11:21:18.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55d8ce81e130 con 0x55d8ce4a0280 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55d8ce82b0c0 con 0x55d8ce8a7ea0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 10 client.0 _ll_register_callbacks cb 0x7f90cc0014b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37322/0 (socket says 172.21.15.136:37322) 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 -- 172.21.15.136:0/1451366796 learned_addr learned my addr 172.21.15.136:0/1451366796 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d97ae640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea2a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea2c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 -- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 msgr2=0x55d8cea2a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea2a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 -- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 msgr2=0x55d8cea2c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea2c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 -- 172.21.15.136:0/1451366796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8ce8a1180 con 0x55d8ce4dfff0 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d97ae640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea2a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d3fff640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c8019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c1fd950cd83fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80b2040 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f90c8003620 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90c4031070 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90c80ce050 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 10 client.4965 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55d8ce4d80c0 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c4031070 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f90c80d1020 con 0x55d8ce4dfff0 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 10 client.4965 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 10 client.4965 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 20 client.4965 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4dde80 0x55d8cea2acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 4 client.4965 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 --> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] -- command(tid 0: {"prefix": "status"}) -- 0x55d8ce82ca30 con 0x55d8ce4dde80 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4dde80 0x55d8cea2acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90d8fad640 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4dde80 0x55d8cea2acd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=311d28761e22c9c9 server_cookie=8d68efbd1305603a in_seq=0 out_seq=0 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 10 client.4965 ms_handle_connect on v2:172.21.15.157:6834/694883637 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 1 -- 172.21.15.136:0/1451366796 <== mds.1 v2:172.21.15.157:6834/694883637 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d8ce82ca30 con 0x55d8ce4dde80 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90b37fe640 10 client.4965 handle_command_reply: tid=0 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 client.4965 shutdown 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 msgr2=0x55d8cea29e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ab0 tx=0x7f90c8019810 comp rx=0 tx=0).stop 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 shutdown_connections 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.679+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:6834/694883637,v1:172.21.15.157:6836/694883637] conn(0x55d8ce4dde80 0x55d8cea2acd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55d8ce8a7ea0 0x55d8cea2a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d8ce4dfff0 0x55d8cea29e90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 --2- 172.21.15.136:0/1451366796 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d8ce4a0280 0x55d8cea2c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 >> 172.21.15.136:0/1451366796 conn(0x55d8cea0afa0 msgr2=0x55d8ce9cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90b37fe640 0 client.4965 ms_handle_reset on v2:172.21.15.157:6834/694883637 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 shutdown_connections 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 1 -- 172.21.15.136:0/1451366796 wait complete. 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90e256e000 20 client.4965 trim_cache size 0 max 16384 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 1, 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4297, 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:21:18.684 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 27.822165776999999, 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 19, 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 26, 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 66.233313308999996 2024-04-25T11:21:18.685 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 msgr2=0x7f90c0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 0x7f90c0071940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f90c4005340 tx=0x7f90c4009040 comp rx=0 tx=0).stop 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 msgr2=0x7f90d414ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90d4075160 tx=0x7f90cc00ec10 comp rx=0 tx=0).stop 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 shutdown_connections 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90d4154870 0x7f90d414ed10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90c006f430 0x7f90c0071940 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90d4147fe0 0x7f90d41f0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 --2- 172.21.15.136:0/3901029916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90bc004810 0x7f90d414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 >> 172.21.15.136:0/3901029916 conn(0x7f90d406f9d0 msgr2=0x7f90d4152ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 shutdown_connections 2024-04-25T11:21:18.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.683+0000 7f90db211640 1 -- 172.21.15.136:0/3901029916 wait complete. 2024-04-25T11:21:18.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T11:21:18.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 Processor -- start 2024-04-25T11:21:18.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 -- start start 2024-04-25T11:21:18.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f00d4074100 0x7f00d40c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d40729c0 con 0x7f00d4070690 2024-04-25T11:21:18.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d4072c80 con 0x7f00d415a250 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dc6b5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f00d4072f60 con 0x7f00d4074100 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00da451640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f00d4074100 0x7f00d40c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dac52640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f00d4070690 0x7f00d416adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60662/0 (socket says 172.21.15.136:60662) 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00dac52640 1 -- 172.21.15.136:0/1641449636 learned_addr learned my addr 172.21.15.136:0/1641449636 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00da451640 1 -- 172.21.15.136:0/1641449636 >> v1:172.21.15.157:6789/0 conn(0x7f00d415a250 legacy=0x7f00d406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00da451640 1 -- 172.21.15.136:0/1641449636 >> v1:172.21.15.136:6789/0 conn(0x7f00d4070690 legacy=0x7f00d416adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00da451640 1 -- 172.21.15.136:0/1641449636 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d4073240 con 0x7f00d4074100 2024-04-25T11:21:18.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.847+0000 7f00d944f640 1 -- 172.21.15.136:0/1641449636 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934693915 0 0) 0x7f00d40729c0 con 0x7f00d4070690 2024-04-25T11:21:18.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00da451640 1 --2- 172.21.15.136:0/1641449636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4074100 0x7f00d40c7c50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f00c40097c0 tx=0x7f00c401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33248170f6caf56c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00d944f640 1 -- 172.21.15.136:0/1641449636 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40b5070 con 0x7f00d4074100 2024-04-25T11:21:18.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00d944f640 1 -- 172.21.15.136:0/1641449636 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c40a8990 con 0x7f00d4074100 2024-04-25T11:21:18.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00d944f640 1 -- 172.21.15.136:0/1641449636 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40a8d60 con 0x7f00d4074100 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1641449636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4074100 msgr2=0x7f00d40c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1641449636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4074100 0x7f00d40c7c50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f00c40097c0 tx=0x7f00c401ec90 comp rx=0 tx=0).stop 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1641449636 shutdown_connections 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1641449636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4074100 0x7f00d40c7c50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1641449636 >> 172.21.15.136:0/1641449636 conn(0x7f00d40c9d50 msgr2=0x7f00d40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1641449636 shutdown_connections 2024-04-25T11:21:18.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.851+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1641449636 wait complete. 2024-04-25T11:21:18.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 Processor -- start 2024-04-25T11:21:18.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- start start 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00d4074100 0x7f00d41557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00d415a250 0x7f00d4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f00d40712c0 con 0x7f00d4070690 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4073800 con 0x7f00d415a250 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00d40c8190 con 0x7f00d4074100 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37332/0 (socket says 172.21.15.136:37332) 2024-04-25T11:21:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 -- 172.21.15.136:0/1214379175 learned_addr learned my addr 172.21.15.136:0/1214379175 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 -- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00d4074100 msgr2=0x7f00d41557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00d9c50640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00d4074100 0x7f00d41557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00d4074100 0x7f00d41557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 -- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00d415a250 msgr2=0x7f00d4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dac52640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00d415a250 0x7f00d4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00d415a250 0x7f00d4155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 -- 172.21.15.136:0/1214379175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d40c5b70 con 0x7f00d4070690 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00da451640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f00c401b8a0 tx=0x7f00c401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00caffd640 1 -- 172.21.15.136:0/1214379175 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40b5040 con 0x7f00d4070690 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00caffd640 1 -- 172.21.15.136:0/1214379175 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c40bf070 con 0x7f00d4070690 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00caffd640 1 -- 172.21.15.136:0/1214379175 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40b0a60 con 0x7f00d4070690 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d41fa3f0 con 0x7f00d4070690 2024-04-25T11:21:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.855+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d41fa9b0 con 0x7f00d4070690 2024-04-25T11:21:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00caffd640 1 -- 172.21.15.136:0/1214379175 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f00c40b0c00 con 0x7f00d4070690 2024-04-25T11:21:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00caffd640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 0x7f00b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00caffd640 1 -- 172.21.15.136:0/1214379175 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f00c412cbb0 con 0x7f00d4070690 2024-04-25T11:21:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00d9c50640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 0x7f00b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 Processor -- start 2024-04-25T11:21:18.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 -- start start 2024-04-25T11:21:18.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00d9c50640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 0x7f00b8071910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f00d4144bd0 tx=0x7f00d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8165d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5615dc693160 con 0x5615dc2aa280 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5615dc628e20 con 0x5615dc6b1ea0 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5615dc5e1a20 con 0x5615dc6c5c20 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37338/0 (socket says 172.21.15.136:37338) 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 -- 172.21.15.136:0/3640692757 learned_addr learned my addr 172.21.15.136:0/3640692757 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00da451640 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8165d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00d9c50640 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 -- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 msgr2=0x5615dc8165d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8165d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 -- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 msgr2=0x5615dc7e1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.859+0000 7f00dac52640 1 -- 172.21.15.136:0/3640692757 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615dc5e6100 con 0x5615dc2aa280 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00dac52640 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00cc001d70 tx=0x7f00cc00d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36088a79fd8f249 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00b7fff640 1 -- 172.21.15.136:0/3640692757 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00d980 con 0x5615dc2aa280 2024-04-25T11:21:18.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00b7fff640 1 -- 172.21.15.136:0/3640692757 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00cc00db20 con 0x5615dc2aa280 2024-04-25T11:21:18.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00b7fff640 1 -- 172.21.15.136:0/3640692757 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc0070d0 con 0x5615dc2aa280 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- 172.21.15.136:0/3640692757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 msgr2=0x5615dc7e1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00cc001d70 tx=0x7f00cc00d150 comp rx=0 tx=0).stop 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- 172.21.15.136:0/3640692757 shutdown_connections 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc7e1a90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- 172.21.15.136:0/3640692757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- 172.21.15.136:0/3640692757 >> 172.21.15.136:0/3640692757 conn(0x5615dc2e9ff0 msgr2=0x5615dc819030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- 172.21.15.136:0/3640692757 shutdown_connections 2024-04-25T11:21:18.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- 172.21.15.136:0/3640692757 wait complete. 2024-04-25T11:21:18.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- start start 2024-04-25T11:21:18.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 Processor -- start 2024-04-25T11:21:18.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc2aa280 0x5615dc7e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8208f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5615dc636220 con 0x5615dc6b1ea0 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5615dc5ccea0 con 0x5615dc2aa280 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5615dc623b00 con 0x5615dc6c5c20 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00e3a12000 10 client.0 _ll_register_callbacks cb 0x5615dc635dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00d9c50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00d9c50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37354/0 (socket says 172.21.15.136:37354) 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00d9c50640 1 -- 172.21.15.136:0/761176167 learned_addr learned my addr 172.21.15.136:0/761176167 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.863+0000 7f00da451640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc2aa280 0x5615dc7e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00dac52640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8208f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 -- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 msgr2=0x5615dc8208f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8208f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 -- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc2aa280 msgr2=0x5615dc7e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc2aa280 0x5615dc7e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 -- 172.21.15.136:0/761176167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615dc5c8500 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00d9c50640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00d00038a0 tx=0x7f00d0001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dae6170d0476af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d000dbd0 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f00d000dd70 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00cc00db20 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f00d0007050 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 10 client.4980 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5615dc5aeea0 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00cc00db20 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f00d002a050 con 0x5615dc6b1ea0 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 10 client.4980 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 10 client.4980 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 20 client.4980 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2ce7d0 0x5615dc821630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 4 client.4980 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T11:21:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 --> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615dc5c1be0 con 0x5615dc2ce7d0 2024-04-25T11:21:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00da451640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2ce7d0 0x5615dc821630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00da451640 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2ce7d0 0x5615dc821630 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80777b5e46ad127d server_cookie=f231572041dc9b76 in_seq=0 out_seq=0 2024-04-25T11:21:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 10 client.4980 ms_handle_connect on v2:172.21.15.157:6838/1671830331 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 1 -- 172.21.15.136:0/761176167 <== mds.0 v2:172.21.15.157:6838/1671830331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5615dc5c1be0 con 0x5615dc2ce7d0 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00b57fa640 10 client.4980 handle_command_reply: tid=0 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.867+0000 7f00e3a12000 1 client.4980 shutdown 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 msgr2=0x5615dc7e1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00d00038a0 tx=0x7f00d0001090 comp rx=0 tx=0).stop 2024-04-25T11:21:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 shutdown_connections 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2ce7d0 0x5615dc821630 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc6c5c20 0x5615dc8208f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc6b1ea0 0x5615dc7e1a20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00b57fa640 0 client.4980 ms_handle_reset on v2:172.21.15.157:6838/1671830331 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 --2- 172.21.15.136:0/761176167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc2aa280 0x5615dc7e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 >> 172.21.15.136:0/761176167 conn(0x5615dc2e9ff0 msgr2=0x5615dc8181e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 shutdown_connections 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 1 -- 172.21.15.136:0/761176167 wait complete. 2024-04-25T11:21:18.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.871+0000 7f00e3a12000 20 client.4980 trim_cache size 0 max 16384 2024-04-25T11:21:18.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.887+0000 7f00e3a12000 1 Processor -- start 2024-04-25T11:21:18.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.887+0000 7f00e3a12000 1 -- start start 2024-04-25T11:21:18.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 0x5615dc8881d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc8360e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f00d4204c10 con 0x5615dc2ce7d0 2024-04-25T11:21:18.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f00d414e400 con 0x5615dc6b1ea0 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5615dc5be7d0 con 0x5615dc2aa280 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37356/0 (socket says 172.21.15.136:37356) 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 -- 172.21.15.136:0/718850630 learned_addr learned my addr 172.21.15.136:0/718850630 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00da451640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 0x5615dc8881d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00dac52640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc8360e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 -- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 msgr2=0x5615dc8881d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 0x5615dc8881d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 -- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 msgr2=0x5615dc8360e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc8360e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 -- 172.21.15.136:0/718850630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615dc5e1c70 con 0x5615dc2ce7d0 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00da451640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 0x5615dc8881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00d9c50640 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f00d0000f90 tx=0x7f00d0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87aeefefbd90141 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00b77fe640 1 -- 172.21.15.136:0/718850630 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0031030 con 0x5615dc2ce7d0 2024-04-25T11:21:18.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00b77fe640 1 -- 172.21.15.136:0/718850630 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d0019040 con 0x5615dc2ce7d0 2024-04-25T11:21:18.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00b77fe640 1 -- 172.21.15.136:0/718850630 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d000d820 con 0x5615dc2ce7d0 2024-04-25T11:21:18.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- 172.21.15.136:0/718850630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 msgr2=0x5615dc888710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f00d0000f90 tx=0x7f00d0014000 comp rx=0 tx=0).stop 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- 172.21.15.136:0/718850630 shutdown_connections 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc8360e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2ce7d0 0x5615dc888710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 --2- 172.21.15.136:0/718850630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2aa280 0x5615dc8881d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- 172.21.15.136:0/718850630 >> 172.21.15.136:0/718850630 conn(0x5615dc2e9ff0 msgr2=0x5615dc820820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- 172.21.15.136:0/718850630 shutdown_connections 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- 172.21.15.136:0/718850630 wait complete. 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 -- start start 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.891+0000 7f00e3a12000 1 Processor -- start 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2ce7d0 0x5615dc893210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc852a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5615dc62d450 con 0x5615dc2aa280 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5615dc628130 con 0x5615dc6b1ea0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5615dc6350c0 con 0x5615dc2ce7d0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 10 client.0 _ll_register_callbacks cb 0x7f00d0001250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37370/0 (socket says 172.21.15.136:37370) 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 -- 172.21.15.136:0/243151005 learned_addr learned my addr 172.21.15.136:0/243151005 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00d9c50640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2ce7d0 0x5615dc893210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dac52640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc852a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 -- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2ce7d0 msgr2=0x5615dc893210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2ce7d0 0x5615dc893210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 -- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 msgr2=0x5615dc852a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc852a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 -- 172.21.15.136:0/243151005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615dc6ab180 con 0x5615dc2aa280 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00da451640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f00c40f5630 tx=0x7f00c40a8990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d36f512b17fcd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:18.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40f9040 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00d0019040 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mon.0 v2:172.21.15.136:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f00c4101980 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f00c40fd3f0 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 10 client.4986 fetch_fsmap learned FSMap version 19 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5615dc2e20c0 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mon.0 v2:172.21.15.136:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c4119e50 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mon.0 v2:172.21.15.136:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f00c40b8050 con 0x5615dc2aa280 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 10 client.4986 fetch_fsmap finished waiting for FSMap version 19 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 10 client.4986 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 20 client.4986 populate_metadata read hostname 'smithi136' 2024-04-25T11:21:18.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2e7e80 0x5615dc89bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 4 client.4986 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 --> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] -- command(tid 0: {"prefix": "status"}) -- 0x5615dc636a30 con 0x5615dc2e7e80 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00d9c50640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2e7e80 0x5615dc89bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 10 client.4986 ms_handle_connect on v2:172.21.15.157:6838/1671830331 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00d9c50640 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2e7e80 0x5615dc89bb40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=583a21bb401213cc server_cookie=358af22a1416385c in_seq=0 out_seq=0 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 1 -- 172.21.15.136:0/243151005 <== mds.0 v2:172.21.15.157:6838/1671830331 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5615dc636a30 con 0x5615dc2e7e80 2024-04-25T11:21:18.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 10 client.4986 handle_command_reply: tid=0 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 client.4986 shutdown 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 msgr2=0x5615dc892cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f00c40f5630 tx=0x7f00c40a8990 comp rx=0 tx=0).stop 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 shutdown_connections 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615dc6b1ea0 0x5615dc852a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:6838/1671830331,v1:172.21.15.157:6839/1671830331] conn(0x5615dc2e7e80 0x5615dc89bb40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5615dc2ce7d0 0x5615dc893210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 --2- 172.21.15.136:0/243151005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5615dc2aa280 0x5615dc892cd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00b57fa640 0 client.4986 ms_handle_reset on v2:172.21.15.157:6838/1671830331 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 >> 172.21.15.136:0/243151005 conn(0x5615dc2e9ff0 msgr2=0x5615dc81f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 shutdown_connections 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 1 -- 172.21.15.136:0/243151005 wait complete. 2024-04-25T11:21:18.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00e3a12000 20 client.4986 trim_cache size 0 max 16384 2024-04-25T11:21:18.902 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:18.902 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_fsid": "6dead98d-43fa-4435-a3fe-cc7c15301a50", 2024-04-25T11:21:18.902 INFO:teuthology.orchestra.run.smithi136.stdout: "whoami": 0, 2024-04-25T11:21:18.902 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4303, 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "want_state": "up:active", 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "state": "up:active", 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs", 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "rank_uptime": 58.867819679999997, 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "mdsmap_epoch": 19, 2024-04-25T11:21:18.903 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch": 26, 2024-04-25T11:21:18.904 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_epoch_barrier": 26, 2024-04-25T11:21:18.904 INFO:teuthology.orchestra.run.smithi136.stdout: "uptime": 66.446397357999999 2024-04-25T11:21:18.904 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:18.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 msgr2=0x7f00b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 0x7f00b8071910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f00d4144bd0 tx=0x7f00d0009040 comp rx=0 tx=0).stop 2024-04-25T11:21:18.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 msgr2=0x7f00d4143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:18.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f00c401b8a0 tx=0x7f00c401ba30 comp rx=0 tx=0).stop 2024-04-25T11:21:18.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 shutdown_connections 2024-04-25T11:21:18.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00d415a250 0x7f00d4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f00b806f400 0x7f00b8071910 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00d4074100 0x7f00d41557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 --2- 172.21.15.136:0/1214379175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00d4070690 0x7f00d4143d30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:18.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 >> 172.21.15.136:0/1214379175 conn(0x7f00d40c9d50 msgr2=0x7f00d406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:18.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.895+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 shutdown_connections 2024-04-25T11:21:18.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:18.899+0000 7f00dc6b5640 1 -- 172.21.15.136:0/1214379175 wait complete. 2024-04-25T11:21:18.915 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T11:21:18.928 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T11:21:18.929 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T11:21:18.929 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T11:21:18.929 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T11:21:18.945 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T11:21:18.946 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-25T11:21:18.946 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-25T11:21:18.946 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T11:21:18.946 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:18.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:18.946 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list 2024-04-25T11:21:18.954 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T11:21:18.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:18.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T11:21:19.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link delete ceph-brx 2024-04-25T11:21:19.222 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T11:21:19.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T11:21:19.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.379+0000 7fbf0840c640 1 Processor -- start 2024-04-25T11:21:19.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.379+0000 7fbf0840c640 1 -- start start 2024-04-25T11:21:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf0840c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 0x7fbf0015a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf0840c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf0840c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf000729c0 con 0x7fbf00074100 2024-04-25T11:21:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf0840c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbf0015aba0 con 0x7fbf0015a250 2024-04-25T11:21:19.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf0840c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbf00072c80 con 0x7fbf0006ff80 2024-04-25T11:21:19.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf059a7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 0x7fbf0015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf059a7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 0x7fbf0015a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:57084/0 (socket says 172.21.15.136:57084) 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:43292/0 (socket says 172.21.15.136:43292) 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf059a7640 1 -- 172.21.15.136:0/2580672159 learned_addr learned my addr 172.21.15.136:0/2580672159 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 -- 172.21.15.136:0/2580672159 >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 msgr2=0x7fbf0015a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 --2- 172.21.15.136:0/2580672159 >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 0x7fbf0015a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 -- 172.21.15.136:0/2580672159 >> v1:172.21.15.157:6789/0 conn(0x7fbf00074100 legacy=0x7fbf000c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.383+0000 7fbf069a9640 1 -- 172.21.15.136:0/2580672159 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf00072f60 con 0x7fbf0006ff80 2024-04-25T11:21:19.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf069a9640 1 --2- 172.21.15.136:0/2580672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87e2f59a6b82c93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:19.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf051a6640 1 -- 172.21.15.136:0/2580672159 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80b4070 con 0x7fbf0006ff80 2024-04-25T11:21:19.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf051a6640 1 -- 172.21.15.136:0/2580672159 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef80af600 con 0x7fbf0006ff80 2024-04-25T11:21:19.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf051a6640 1 -- 172.21.15.136:0/2580672159 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80af9d0 con 0x7fbf0006ff80 2024-04-25T11:21:19.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 -- 172.21.15.136:0/2580672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 msgr2=0x7fbf00070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2580672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef801f150 comp rx=0 tx=0).stop 2024-04-25T11:21:19.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 -- 172.21.15.136:0/2580672159 shutdown_connections 2024-04-25T11:21:19.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2580672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf0006ff80 0x7fbf00070390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2580672159 >> v2:172.21.15.157:3300/0 conn(0x7fbf0015a250 0x7fbf0015a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 -- 172.21.15.136:0/2580672159 >> 172.21.15.136:0/2580672159 conn(0x7fbf000c9d50 msgr2=0x7fbf000ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:19.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 -- 172.21.15.136:0/2580672159 shutdown_connections 2024-04-25T11:21:19.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 -- 172.21.15.136:0/2580672159 wait complete. 2024-04-25T11:21:19.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.387+0000 7fbf0840c640 1 Processor -- start 2024-04-25T11:21:19.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- start start 2024-04-25T11:21:19.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 0x7fbf001631b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf00074100 0x7fbf00147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbf00073800 con 0x7fbf0015a250 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbf00073520 con 0x7fbf0006ff80 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbf000712c0 con 0x7fbf00074100 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37382/0 (socket says 172.21.15.136:37382) 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 -- 172.21.15.136:0/2343467015 learned_addr learned my addr 172.21.15.136:0/2343467015 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf059a7640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf00074100 0x7fbf00147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf061a8640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 0x7fbf001631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 -- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf00074100 msgr2=0x7fbf00147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf00074100 0x7fbf00147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 -- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 msgr2=0x7fbf001631b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 0x7fbf001631b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 -- 172.21.15.136:0/2343467015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf001636f0 con 0x7fbf0015a250 2024-04-25T11:21:19.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf061a8640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 0x7fbf001631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf069a9640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80b4040 con 0x7fbf0015a250 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef80c1070 con 0x7fbf0015a250 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf00164680 con 0x7fbf0015a250 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8009450 con 0x7fbf0015a250 2024-04-25T11:21:19.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf001ec470 con 0x7fbf0015a250 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.391+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0006a350 con 0x7fbf0015a250 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.395+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbef80ba6b0 con 0x7fbf0015a250 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.395+0000 7fbef2ffd640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 0x7fbee4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.395+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fbef812df30 con 0x7fbf0015a250 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.395+0000 7fbf061a8640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 0x7fbee4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:19.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.395+0000 7fbf061a8640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 0x7fbee4071910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbef4001670 tx=0x7fbef4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:19.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.399+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbef8018cb0 con 0x7fbf0015a250 2024-04-25T11:21:19.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:21:19.510 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:21:19.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.595+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbf000c4250 con 0x7fbf0015a250 2024-04-25T11:21:19.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbef2ffd640 1 -- 172.21.15.136:0/2343467015 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+88 (secure 0 0 0) 0x7fbef80faab0 con 0x7fbf0015a250 2024-04-25T11:21:19.596 INFO:teuthology.orchestra.run.smithi136.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 msgr2=0x7fbee4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 0x7fbee4071910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbef4001670 tx=0x7fbef4009040 comp rx=0 tx=0).stop 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 msgr2=0x7fbf00147ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009d90 tx=0x7fbef801bef0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 shutdown_connections 2024-04-25T11:21:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf0015a250 0x7fbf00147ab0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbee406f400 0x7fbee4071910 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf00074100 0x7fbf00147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 --2- 172.21.15.136:0/2343467015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf0006ff80 0x7fbf001631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 >> 172.21.15.136:0/2343467015 conn(0x7fbf000c9d50 msgr2=0x7fbf00157bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 shutdown_connections 2024-04-25T11:21:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19.599+0000 7fbf0840c640 1 -- 172.21.15.136:0/2343467015 wait complete. 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi136.front.sepia.ceph.com 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T11:21:19.617 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T11:21:19.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.618 DEBUG:teuthology.orchestra.run.smithi136:> ip addr 2024-04-25T11:21:19.623 INFO:teuthology.orchestra.run.smithi136.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 ::1/128 scope host 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1a brd ff:ff:ff:ff:ff:ff 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f0 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:d9:3b:1b brd ff:ff:ff:ff:ff:ff 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: altname enp7s0f1 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T11:21:19.624 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:42 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 0c:c4:7a:88:6e:43 brd ff:ff:ff:ff:ff:ff 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.21.15.136/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft 42450sec preferred_lft 42450sec 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: link/ether 02:42:f7:de:2c:ef brd ff:ff:ff:ff:ff:ff 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-25T11:21:19.625 INFO:teuthology.orchestra.run.smithi136.stdout: valid_lft forever preferred_lft forever 2024-04-25T11:21:19.625 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link add name ceph-brx type bridge 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip addr flush dev ceph-brx 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set ceph-brx up 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T11:21:19.626 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:21:19.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:21:19.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:19.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.763 DEBUG:teuthology.orchestra.run.smithi136:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T11:21:19.812 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-04-25T11:21:19.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.813 DEBUG:teuthology.orchestra.run.smithi136:> route 2024-04-25T11:21:19.864 INFO:teuthology.orchestra.run.smithi136.stdout:Kernel IP routing table 2024-04-25T11:21:19.864 INFO:teuthology.orchestra.run.smithi136.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T11:21:19.864 INFO:teuthology.orchestra.run.smithi136.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T11:21:19.865 INFO:teuthology.orchestra.run.smithi136.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T11:21:19.865 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:21:19.865 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:21:19.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T11:21:19.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T11:21:19.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T11:21:19.866 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:21:19.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:21:19.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:19.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.985 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list 2024-04-25T11:21:19.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:19.990 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list-id 2024-04-25T11:21:20.040 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:21:20.040 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:21:20.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:20.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T11:21:20.041 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:21:20.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:21:20.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:20.152 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T11:21:20.153 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:21:20.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:21:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:20.294 DEBUG:teuthology.orchestra.run.smithi136:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T11:21:20.294 DEBUG:teuthology.orchestra.run.smithi136:> set -e 2024-04-25T11:21:20.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set brx.0 up 2024-04-25T11:21:20.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T11:21:20.294 DEBUG:teuthology.orchestra.run.smithi136:> ') 2024-04-25T11:21:20.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T11:21:20.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:20.392 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T11:21:20.392 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:21:20.392 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:20.440 INFO:teuthology.orchestra.run.smithi136.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T11:21:20.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T11:21:20.440 DEBUG:teuthology.orchestra.run.smithi136:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:20.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:21:20.549 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:21:20.597 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:21:20.598 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:21:20.599 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:21:20.600 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:21:20.600 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:21:20.600 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:21:20.600 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:20.600 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:20.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:20.601 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:21:20.649 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T11:21:20.649 DEBUG:teuthology.orchestra.run.smithi136:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-25T11:21:20.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:21:20.712 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:21:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:21:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:21:20.763 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:21:20.763 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:21:20.763 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:21:20.763 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:20.763 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:20.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:20.764 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:21:20.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi136.stderr:2024-04-25T11:21:20.811+0000 7fb434338580 -1 init, newargv = 0x7fb424005d30 newargc=15 2024-04-25T11:21:20.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi136.stderr:ceph-fuse[35981]: starting ceph client 2024-04-25T11:21:20.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi136.stderr:ceph-fuse[35981]: starting fuse 2024-04-25T11:21:21.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo modprobe fuse 2024-04-25T11:21:21.930 DEBUG:teuthology.orchestra.run.smithi136:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T11:21:21.979 INFO:teuthology.orchestra.run.smithi136.stdout:/sys 2024-04-25T11:21:21.979 INFO:teuthology.orchestra.run.smithi136.stdout:/proc 2024-04-25T11:21:21.979 INFO:teuthology.orchestra.run.smithi136.stdout:/dev 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/pts 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/run 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/ 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/security 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/shm 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/run/lock 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/cgroup 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/pstore 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/bpf 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/hugepages 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/dev/mqueue 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/debug 2024-04-25T11:21:21.980 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/tracing 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/proc/fs/nfsd 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/fs/fuse/connections 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/sys/kernel/config 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/rpc_pipefs 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/user/1000 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:21:21.981 INFO:teuthology.orchestra.run.smithi136.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:21.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:21.982 DEBUG:teuthology.orchestra.run.smithi136:> ls /sys/fs/fuse/connections 2024-04-25T11:21:22.033 INFO:teuthology.orchestra.run.smithi136.stdout:45 2024-04-25T11:21:22.033 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo stdin-killer -- python3 -c ' 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> import glob 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> import re 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> import os 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> import subprocess 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> def _find_admin_socket(client_name): 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> files = glob.glob(asok_path) 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> # Given a non-glob path, it better be there 2024-04-25T11:21:22.034 DEBUG:teuthology.orchestra.run.smithi136:> if "*" not in asok_path: 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> assert(len(files) == 1) 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> return files[0] 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> for f in files: 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> contents = proc_f.read() 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> if mountpoint in contents: 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> return f 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> print(_find_admin_socket("client.0")) 2024-04-25T11:21:22.035 DEBUG:teuthology.orchestra.run.smithi136:> ' 2024-04-25T11:21:22.162 INFO:teuthology.orchestra.run.smithi136.stdout:/var/run/ceph/ceph-client.0.35981.asok 2024-04-25T11:21:22.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T11:21:22.174 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.35981.asok 2024-04-25T11:21:22.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:22.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.35981.asok status 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "metadata": { 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "entity_id": "0", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "hostname": "smithi136", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "pid": "35981", 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "root": "/" 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "dentry_count": 0, 2024-04-25T11:21:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "dentry_pinned_count": 0, 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "id": 4995, 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "inst": { 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "name": { 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "client", 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "num": 4995 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": { 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "v1", 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": "192.168.0.1:0", 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "nonce": 3752176696 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": { 2024-04-25T11:21:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "v1", 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "addr": "192.168.0.1:0", 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "nonce": 3752176696 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "inst_str": "client.4995 192.168.0.1:0/3752176696", 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "addr_str": "192.168.0.1:0/3752176696", 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "inode_count": 1, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_epoch": 19, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_epoch": 26, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_epoch_barrier": 0, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "blocklisted": false, 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "fs_name": "cephfs" 2024-04-25T11:21:22.350 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:21:22.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:22.364 DEBUG:teuthology.orchestra.run.smithi136:> stat --file-system '--printf=%T 2024-04-25T11:21:22.364 DEBUG:teuthology.orchestra.run.smithi136:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:22.370 INFO:teuthology.orchestra.run.smithi136.stdout:fuseblk 2024-04-25T11:21:22.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:22.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:21:22.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:22.424 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-25T11:21:22.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:22.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 Processor -- start 2024-04-25T11:21:22.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 -- start start 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7f0c15cb70 0x7f7f0c167a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c0c6470 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c156790 con 0x7f7f0c156380 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f14480640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0c0cd390 con 0x7f7f0c15cb70 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1221c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7f0c0c66e0 0x7f7f0c15f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60670/0 (socket says 172.21.15.136:60670) 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1221c640 1 -- 172.21.15.136:0/3684578642 learned_addr learned my addr 172.21.15.136:0/3684578642 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:22.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3684578642 >> v2:172.21.15.157:3300/0 conn(0x7f7f0c15cb70 0x7f7f0c167a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515551056 0 0) 0x7f7f0c0c6470 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ef8003530 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2616132223 0 0) 0x7f7ef8003530 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 >> v2:172.21.15.157:3300/0 conn(0x7f7f0c15cb70 msgr2=0x7f7f0c167a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 --2- 172.21.15.136:0/3684578642 >> v2:172.21.15.157:3300/0 conn(0x7f7f0c15cb70 0x7f7f0c167a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 >> v1:172.21.15.157:6789/0 conn(0x7f7f0c156380 legacy=0x7f7f0c0cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.623+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c15c0b0 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f7f080016a0 con 0x7f7f0c0c66e0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 >> v1:172.21.15.136:6789/0 conn(0x7f7f0c0c66e0 legacy=0x7f7f0c15f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef80039c0 0x7f7ef8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef8004810 0x7f7ef8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7ef800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef800c140 con 0x7f7ef80051d0 2024-04-25T11:21:22.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef800c2f0 con 0x7f7ef80039c0 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef800c4f0 con 0x7f7ef8004810 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7ef800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1221c640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef80039c0 0x7f7ef8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef8004810 0x7f7ef8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 -- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef8004810 msgr2=0x7f7ef8004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef8004810 0x7f7ef8004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 -- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef80039c0 msgr2=0x7f7ef8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ef80039c0 0x7f7ef8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 -- 172.21.15.136:0/3684578642 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef8003530 con 0x7f7ef80051d0 2024-04-25T11:21:22.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ef8004810 0x7f7ef8004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:22.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 -- 172.21.15.136:0/3684578642 reap_dead start 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7ef800b9d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009f90 tx=0x7f7f0401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f70bce82766d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f040b2070 con 0x7f7ef80051d0 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f040a5bb0 con 0x7f7ef80051d0 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f1121a640 1 -- 172.21.15.136:0/3684578642 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04009c10 con 0x7f7ef80051d0 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 -- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 msgr2=0x7f7ef800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7ef800b9d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009f90 tx=0x7f7f0401cbc0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 -- 172.21.15.136:0/3684578642 shutdown_connections 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 --2- 172.21.15.136:0/3684578642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7ef800b9d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 -- 172.21.15.136:0/3684578642 >> 172.21.15.136:0/3684578642 conn(0x7f7f0c06d6c0 msgr2=0x7f7f0c06e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 -- 172.21.15.136:0/3684578642 shutdown_connections 2024-04-25T11:21:22.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.627+0000 7f7f14480640 1 -- 172.21.15.136:0/3684578642 wait complete. 2024-04-25T11:21:22.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 Processor -- start 2024-04-25T11:21:22.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- start start 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 0x7f7f0c157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 0x7f7f0c0cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c15c370 con 0x7f7ef80051d0 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c15c6e0 con 0x7f7f0c0cb8a0 2024-04-25T11:21:22.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c15cf80 con 0x7f7f0c0c7460 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37412/0 (socket says 172.21.15.136:37412) 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 -- 172.21.15.136:0/3596030137 learned_addr learned my addr 172.21.15.136:0/3596030137 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 0x7f7f0c0cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 0x7f7f0c157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 -- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 msgr2=0x7f7f0c0cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 0x7f7f0c0cac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 -- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 msgr2=0x7f7f0c157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 0x7f7f0c157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 -- 172.21.15.136:0/3596030137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c1f25b0 con 0x7f7ef80051d0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f12a1d640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 0x7f7f0c0cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 0x7f7f0c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f1221c640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e780 tx=0x7f7f0800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800c6c0 con 0x7f7ef80051d0 2024-04-25T11:21:22.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f08018070 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800cae0 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c1f2890 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.631+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c1f2e50 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c06a350 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7f0800cc80 con 0x7f7ef80051d0 2024-04-25T11:21:22.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f02ffd640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 0x7f7ef0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7f0808ff20 con 0x7f7ef80051d0 2024-04-25T11:21:22.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 0x7f7ef0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.635+0000 7f7f11a1b640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 0x7f7ef0071910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0cd490 tx=0x7f7efc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.639+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7f0805cb80 con 0x7f7ef80051d0 2024-04-25T11:21:22.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.803+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f0c0cda30 con 0x7f7ef80051d0 2024-04-25T11:21:22.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.803+0000 7f7f02ffd640 1 -- 172.21.15.136:0/3596030137 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7f7f0805c480 con 0x7f7ef80051d0 2024-04-25T11:21:22.803 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:22.803 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:22.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 msgr2=0x7f7ef0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 0x7f7ef0071910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0cd490 tx=0x7f7efc009040 comp rx=0 tx=0).stop 2024-04-25T11:21:22.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 msgr2=0x7f7f0c0cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e780 tx=0x7f7f0800ec40 comp rx=0 tx=0).stop 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 shutdown_connections 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ef006f400 0x7f7ef0071910 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7f0c0c7460 0x7f7f0c0cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f0c0cb8a0 0x7f7f0c157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 --2- 172.21.15.136:0/3596030137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef80051d0 0x7f7f0c0cf350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 >> 172.21.15.136:0/3596030137 conn(0x7f7f0c06d6c0 msgr2=0x7f7f0c0b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 shutdown_connections 2024-04-25T11:21:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.807+0000 7f7f14480640 1 -- 172.21.15.136:0/3596030137 wait complete. 2024-04-25T11:21:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:21:22.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:21:22.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 Processor -- start 2024-04-25T11:21:22.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 -- start start 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7b8073590 0x7fc7b80739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b815d590 con 0x7fc7b8072dd0 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b815d870 con 0x7fc7b80729c0 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.975+0000 7fc7c017e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc7b815d340 con 0x7fc7b8073590 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7bd719640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc7b8072dd0 0x7fc7b80c7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60680/0 (socket says 172.21.15.136:60680) 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7bd719640 1 -- 172.21.15.136:0/3263009433 learned_addr learned my addr 172.21.15.136:0/3263009433 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7be71b640 1 --2- 172.21.15.136:0/3263009433 >> v2:172.21.15.136:3300/0 conn(0x7fc7b8073590 0x7fc7b80739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7be71b640 1 -- 172.21.15.136:0/3263009433 >> v1:172.21.15.157:6789/0 conn(0x7fc7b80729c0 legacy=0x7fc7b8142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:22.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7be71b640 1 -- 172.21.15.136:0/3263009433 >> v1:172.21.15.136:6789/0 conn(0x7fc7b8072dd0 legacy=0x7fc7b80c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7be71b640 1 -- 172.21.15.136:0/3263009433 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b815db50 con 0x7fc7b8073590 2024-04-25T11:21:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7be71b640 1 --2- 172.21.15.136:0/3263009433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b8073590 0x7fc7b80739a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009d90 tx=0x7fc7b001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34687f949f7e977b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7bcf18640 1 -- 172.21.15.136:0/3263009433 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00b2070 con 0x7fc7b8073590 2024-04-25T11:21:22.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7bcf18640 1 -- 172.21.15.136:0/3263009433 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b00a57f0 con 0x7fc7b8073590 2024-04-25T11:21:22.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7bcf18640 1 -- 172.21.15.136:0/3263009433 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00a5bc0 con 0x7fc7b8073590 2024-04-25T11:21:22.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 -- 172.21.15.136:0/3263009433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b8073590 msgr2=0x7fc7b80739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 --2- 172.21.15.136:0/3263009433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b8073590 0x7fc7b80739a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009d90 tx=0x7fc7b001c930 comp rx=0 tx=0).stop 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 -- 172.21.15.136:0/3263009433 shutdown_connections 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 --2- 172.21.15.136:0/3263009433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b8073590 0x7fc7b80739a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 -- 172.21.15.136:0/3263009433 >> 172.21.15.136:0/3263009433 conn(0x7fc7b80c9d50 msgr2=0x7fc7b80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 -- 172.21.15.136:0/3263009433 shutdown_connections 2024-04-25T11:21:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.979+0000 7fc7c017e640 1 -- 172.21.15.136:0/3263009433 wait complete. 2024-04-25T11:21:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 Processor -- start 2024-04-25T11:21:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 -- start start 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7b8072dd0 0x7fc7b81410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7b8073590 0x7fc7b8152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b80c84e0 con 0x7fc7b80729c0 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b80c8360 con 0x7fc7b8072dd0 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7c017e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b8167f00 con 0x7fc7b8073590 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37436/0 (socket says 172.21.15.136:37436) 2024-04-25T11:21:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 -- 172.21.15.136:0/1890072992 learned_addr learned my addr 172.21.15.136:0/1890072992 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7be71b640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7b8073590 0x7fc7b8152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 -- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7b8073590 msgr2=0x7fc7b8152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bd719640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7b8072dd0 0x7fc7b81410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7b8073590 0x7fc7b8152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 -- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7b8072dd0 msgr2=0x7fc7b81410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7b8072dd0 0x7fc7b81410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 -- 172.21.15.136:0/1890072992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b80c94e0 con 0x7fc7b80729c0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7bdf1a640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400b440 tx=0x7fc7b400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4015020 con 0x7fc7b80729c0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b4011890 con 0x7fc7b80729c0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.983+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4011c80 con 0x7fc7b80729c0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b81f9730 con 0x7fc7b80729c0 2024-04-25T11:21:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b8138cf0 con 0x7fc7b80729c0 2024-04-25T11:21:22.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b806a350 con 0x7fc7b80729c0 2024-04-25T11:21:22.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc7b4018020 con 0x7fc7b80729c0 2024-04-25T11:21:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7ae7fc640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 0x7fc79c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc7b408f4f0 con 0x7fc7b80729c0 2024-04-25T11:21:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7bd719640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 0x7fc79c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.987+0000 7fc7bd719640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 0x7fc79c071910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8134cc0 tx=0x7fc7a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:22.991+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc7b401f030 con 0x7fc7b80729c0 2024-04-25T11:21:23.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.151+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7b8074c80 con 0x7fc7b80729c0 2024-04-25T11:21:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.151+0000 7fc7ae7fc640 1 -- 172.21.15.136:0/1890072992 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fc7b400d040 con 0x7fc7b80729c0 2024-04-25T11:21:23.151 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:21:23.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 msgr2=0x7fc79c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 0x7fc79c071910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8134cc0 tx=0x7fc7a8009040 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 msgr2=0x7fc7b81f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc7b400b440 tx=0x7fc7b400b900 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 shutdown_connections 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc79c06f400 0x7fc79c071910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7b8073590 0x7fc7b8152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7b8072dd0 0x7fc7b81410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 --2- 172.21.15.136:0/1890072992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7b80729c0 0x7fc7b81f91f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 >> 172.21.15.136:0/1890072992 conn(0x7fc7b80c9d50 msgr2=0x7fc7b8142cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 shutdown_connections 2024-04-25T11:21:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.155+0000 7fc7c017e640 1 -- 172.21.15.136:0/1890072992 wait complete. 2024-04-25T11:21:23.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:21:23.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 Processor -- start 2024-04-25T11:21:23.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 -- start start 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19b0164cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19b0159900 con 0x7f19b00c7c50 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19b015c3e0 con 0x7f19b00c4750 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b709c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f19b01596d0 con 0x7f19b00c4b60 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b4e38640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f19b00c7c50 0x7f19b00c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60684/0 (socket says 172.21.15.136:60684) 2024-04-25T11:21:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b4e38640 1 -- 172.21.15.136:0/4237164549 learned_addr learned my addr 172.21.15.136:0/4237164549 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:23.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252999657 0 0) 0x7f19b0159900 con 0x7f19b00c7c50 2024-04-25T11:21:23.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b5639640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19b0164cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1998003530 con 0x7f19b00c7c50 2024-04-25T11:21:23.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 930134911 0 0) 0x7f1998003530 con 0x7f19b00c7c50 2024-04-25T11:21:23.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 msgr2=0x7f19b0164cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:23.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19b0164cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 >> v1:172.21.15.157:6789/0 conn(0x7f19b00c4750 legacy=0x7f19b01615c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b015c6c0 con 0x7f19b00c7c50 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f19ac001680 con 0x7f19b00c7c50 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 >> v1:172.21.15.136:6789/0 conn(0x7f19b00c7c50 legacy=0x7f19b00c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19980039f0 0x7f1998003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1998004810 0x7f1998004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19980052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f199800bc10 con 0x7f1998004810 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f199800bdc0 con 0x7f19980039f0 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f199800bfc0 con 0x7f19b00c4b60 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1998004810 0x7f1998004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b4e38640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19980039f0 0x7f1998003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b5639640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19980052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 -- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 msgr2=0x7f19980052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19980052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 -- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19980039f0 msgr2=0x7f1998003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19980039f0 0x7f1998003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19a7fff640 1 -- 172.21.15.136:0/4237164549 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1998003530 con 0x7f1998004810 2024-04-25T11:21:23.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.315+0000 7f19b5639640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19b00c4b60 0x7f19980052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19a7fff640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1998004810 0x7f1998004c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f19b015d360 tx=0x7f19a001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff8c3b68a12ae21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a00013a0 con 0x7f1998004810 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a00052f0 con 0x7f1998004810 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19a77fe640 1 -- 172.21.15.136:0/4237164549 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a0003840 con 0x7f1998004810 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 -- 172.21.15.136:0/4237164549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1998004810 msgr2=0x7f1998004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 --2- 172.21.15.136:0/4237164549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1998004810 0x7f1998004c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f19b015d360 tx=0x7f19a001c3c0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b5639640 1 -- 172.21.15.136:0/4237164549 reap_dead start 2024-04-25T11:21:23.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 -- 172.21.15.136:0/4237164549 shutdown_connections 2024-04-25T11:21:23.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 -- 172.21.15.136:0/4237164549 >> 172.21.15.136:0/4237164549 conn(0x7f19b00c9d50 msgr2=0x7f19b00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:23.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 -- 172.21.15.136:0/4237164549 shutdown_connections 2024-04-25T11:21:23.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.319+0000 7f19b709c640 1 -- 172.21.15.136:0/4237164549 wait complete. 2024-04-25T11:21:23.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 Processor -- start 2024-04-25T11:21:23.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- start start 2024-04-25T11:21:23.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1998004810 0x7f19b01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 0x7f19b014ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f19b015cc80 con 0x7f19b0145cd0 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f19b0159bc0 con 0x7f19b01f2750 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f19b00c4f70 con 0x7f1998004810 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37444/0 (socket says 172.21.15.136:37444) 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 -- 172.21.15.136:0/2567692688 learned_addr learned my addr 172.21.15.136:0/2567692688 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b4e38640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1998004810 0x7f19b01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a7fff640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 0x7f19b014ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 -- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1998004810 msgr2=0x7f19b01554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1998004810 0x7f19b01554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 -- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 msgr2=0x7f19b014ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 0x7f19b014ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 -- 172.21.15.136:0/2567692688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b01f2b60 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a7fff640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 0x7f19b014ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b5639640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f19b0072670 tx=0x7f19a800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8007e40 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a801a070 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8014610 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b0159e10 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b015b370 con 0x7f19b0145cd0 2024-04-25T11:21:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b006a350 con 0x7f19b0145cd0 2024-04-25T11:21:23.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.323+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f19a80147b0 con 0x7f19b0145cd0 2024-04-25T11:21:23.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.327+0000 7f19a57fa640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.327+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f19a808fc10 con 0x7f19b0145cd0 2024-04-25T11:21:23.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.327+0000 7f19b4e38640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.327+0000 7f19b4e38640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 0x7f199c071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f19ac001ec0 tx=0x7f19ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.331+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f19a805c700 con 0x7f19b0145cd0 2024-04-25T11:21:23.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.491+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19b0156590 con 0x7f19b0145cd0 2024-04-25T11:21:23.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.491+0000 7f19a57fa640 1 -- 172.21.15.136:0/2567692688 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f19a805c000 con 0x7f19b0145cd0 2024-04-25T11:21:23.490 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:23.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 msgr2=0x7f199c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 0x7f199c071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f19ac001ec0 tx=0x7f19ac009040 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 msgr2=0x7f19b01460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f19b0072670 tx=0x7f19a800f520 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 shutdown_connections 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19b0145cd0 0x7f19b01460e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19b01f2750 0x7f19b014ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 --2- 172.21.15.136:0/2567692688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1998004810 0x7f19b01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 >> 172.21.15.136:0/2567692688 conn(0x7f19b00c9d50 msgr2=0x7f19b00c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 shutdown_connections 2024-04-25T11:21:23.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.495+0000 7f19b709c640 1 -- 172.21.15.136:0/2567692688 wait complete. 2024-04-25T11:21:23.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:21:23.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.663+0000 7fea866f2640 1 Processor -- start 2024-04-25T11:21:23.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 -- start start 2024-04-25T11:21:23.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea800c4750 0x7fea800c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 0x7fea801639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea80159900 con 0x7fea800c7c50 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fea801596d0 con 0x7fea800c4750 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea866f2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fea80159bc0 con 0x7fea801615c0 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea800c4750 0x7fea800c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea800c4750 0x7fea800c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37456/0 (socket says 172.21.15.136:37456) 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fea800c7c50 0x7fea800c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60692/0 (socket says 172.21.15.136:60692) 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 -- 172.21.15.136:0/3027511182 learned_addr learned my addr 172.21.15.136:0/3027511182 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 -- 172.21.15.136:0/3027511182 >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 msgr2=0x7fea801639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea84c8f640 1 --2- 172.21.15.136:0/3027511182 >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 0x7fea801639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7effd640 1 -- 172.21.15.136:0/3027511182 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1058170955 0 0) 0x7fea80159900 con 0x7fea800c7c50 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 --2- 172.21.15.136:0/3027511182 >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 0x7fea801639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 -- 172.21.15.136:0/3027511182 >> v1:172.21.15.136:6789/0 conn(0x7fea800c7c50 legacy=0x7fea800c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea7f7fe640 1 -- 172.21.15.136:0/3027511182 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8015c3e0 con 0x7fea800c4750 2024-04-25T11:21:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.667+0000 7fea84c8f640 1 --2- 172.21.15.136:0/3027511182 >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 0x7fea801639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:23.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea7f7fe640 1 --2- 172.21.15.136:0/3027511182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c4750 0x7fea800c4b60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea80072060 tx=0x7fea7401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74327f9b6de734e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea7effd640 1 -- 172.21.15.136:0/3027511182 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740b5070 con 0x7fea800c4750 2024-04-25T11:21:23.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea7effd640 1 -- 172.21.15.136:0/3027511182 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea740a8a10 con 0x7fea800c4750 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea7effd640 1 -- 172.21.15.136:0/3027511182 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740a8de0 con 0x7fea800c4750 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- 172.21.15.136:0/3027511182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c4750 msgr2=0x7fea800c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 --2- 172.21.15.136:0/3027511182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c4750 0x7fea800c4b60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fea80072060 tx=0x7fea7401ed10 comp rx=0 tx=0).stop 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- 172.21.15.136:0/3027511182 shutdown_connections 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 --2- 172.21.15.136:0/3027511182 >> v2:172.21.15.157:3300/0 conn(0x7fea801615c0 0x7fea801639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 --2- 172.21.15.136:0/3027511182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c4750 0x7fea800c4b60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- 172.21.15.136:0/3027511182 >> 172.21.15.136:0/3027511182 conn(0x7fea800c9d50 msgr2=0x7fea800ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:23.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- 172.21.15.136:0/3027511182 shutdown_connections 2024-04-25T11:21:23.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- 172.21.15.136:0/3027511182 wait complete. 2024-04-25T11:21:23.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 Processor -- start 2024-04-25T11:21:23.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.671+0000 7fea866f2640 1 -- start start 2024-04-25T11:21:23.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea800c4750 0x7fea8014bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea801615c0 0x7fea80150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fea8015cc80 con 0x7fea800c7c50 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea8015c9a0 con 0x7fea801615c0 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea800c5270 con 0x7fea800c4750 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37462/0 (socket says 172.21.15.136:37462) 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 -- 172.21.15.136:0/218516334 learned_addr learned my addr 172.21.15.136:0/218516334 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea84c8f640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea801615c0 0x7fea80150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 -- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea800c4750 msgr2=0x7fea8014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea800c4750 0x7fea8014bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 -- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea801615c0 msgr2=0x7fea80150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea801615c0 0x7fea80150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 -- 172.21.15.136:0/218516334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8014c410 con 0x7fea800c7c50 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7f7fe640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fea740b3040 tx=0x7fea7401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740c0070 con 0x7fea800c7c50 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea74004230 con 0x7fea800c7c50 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea80153b90 con 0x7fea800c7c50 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740b5040 con 0x7fea800c7c50 2024-04-25T11:21:23.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea801f4820 con 0x7fea800c7c50 2024-04-25T11:21:23.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea8006a350 con 0x7fea800c7c50 2024-04-25T11:21:23.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea740c3070 con 0x7fea800c7c50 2024-04-25T11:21:23.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 0x7fea6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:23.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.675+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fea7412c290 con 0x7fea800c7c50 2024-04-25T11:21:23.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.679+0000 7fea7ffff640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 0x7fea6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:23.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.679+0000 7fea7ffff640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 0x7fea6c071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea800c9d20 tx=0x7fea70007970 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:23.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.683+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fea74018870 con 0x7fea800c7c50 2024-04-25T11:21:23.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.839+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fea80159080 con 0x7fea800c7c50 2024-04-25T11:21:23.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea7cff9640 1 -- 172.21.15.136:0/218516334 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fea740f8e10 con 0x7fea800c7c50 2024-04-25T11:21:23.841 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 msgr2=0x7fea6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 0x7fea6c071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea800c9d20 tx=0x7fea70007970 comp rx=0 tx=0).stop 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 msgr2=0x7fea80155e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fea740b3040 tx=0x7fea7401bab0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 shutdown_connections 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea801615c0 0x7fea80150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea800c7c50 0x7fea80155e60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea6c06f400 0x7fea6c071910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 --2- 172.21.15.136:0/218516334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea800c4750 0x7fea8014bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:23.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 >> 172.21.15.136:0/218516334 conn(0x7fea800c9d50 msgr2=0x7fea80151cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:23.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 shutdown_connections 2024-04-25T11:21:23.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:23.843+0000 7fea866f2640 1 -- 172.21.15.136:0/218516334 wait complete. 2024-04-25T11:21:23.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:21:24.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 Processor -- start 2024-04-25T11:21:24.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 -- start start 2024-04-25T11:21:24.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f980c7c50 0x7f4f980c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4f980c8e40 0x7f4f980c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f98159900 con 0x7f4f980c4c90 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f4f981596d0 con 0x7f4f980c7c50 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9cac9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4f98159bc0 con 0x7f4f980c8e40 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f980c7c50 0x7f4f980c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f96d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4f980c4c90 0x7f4f980c68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60704/0 (socket says 172.21.15.136:60704) 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f96d9d640 1 -- 172.21.15.136:0/2662008476 learned_addr learned my addr 172.21.15.136:0/2662008476 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f8ffff640 1 --2- 172.21.15.136:0/2662008476 >> v2:172.21.15.157:3300/0 conn(0x7f4f980c8e40 0x7f4f980c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9659c640 1 -- 172.21.15.136:0/2662008476 >> v2:172.21.15.157:3300/0 conn(0x7f4f980c8e40 msgr2=0x7f4f980c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9659c640 1 --2- 172.21.15.136:0/2662008476 >> v2:172.21.15.157:3300/0 conn(0x7f4f980c8e40 0x7f4f980c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9659c640 1 -- 172.21.15.136:0/2662008476 >> v1:172.21.15.136:6789/0 conn(0x7f4f980c4c90 legacy=0x7f4f980c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f9659c640 1 -- 172.21.15.136:0/2662008476 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9815c3e0 con 0x7f4f980c7c50 2024-04-25T11:21:24.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.011+0000 7f4f95d9b640 1 -- 172.21.15.136:0/2662008476 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2314854470 0 0) 0x7f4f98159900 con 0x7f4f980c4c90 2024-04-25T11:21:24.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9659c640 1 --2- 172.21.15.136:0/2662008476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c7c50 0x7f4f980c8900 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4f980c4620 tx=0x7f4f8001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d368d4cbc134cd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f95d9b640 1 -- 172.21.15.136:0/2662008476 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800b5070 con 0x7f4f980c7c50 2024-04-25T11:21:24.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f95d9b640 1 -- 172.21.15.136:0/2662008476 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f800a8950 con 0x7f4f980c7c50 2024-04-25T11:21:24.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f95d9b640 1 -- 172.21.15.136:0/2662008476 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800a8d20 con 0x7f4f980c7c50 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2662008476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c7c50 msgr2=0x7f4f980c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2662008476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c7c50 0x7f4f980c8900 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4f980c4620 tx=0x7f4f8001ec50 comp rx=0 tx=0).stop 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2662008476 shutdown_connections 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2662008476 >> v2:172.21.15.157:3300/0 conn(0x7f4f980c8e40 0x7f4f980c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2662008476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c7c50 0x7f4f980c8900 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2662008476 >> 172.21.15.136:0/2662008476 conn(0x7f4f980c9d50 msgr2=0x7f4f980ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2662008476 shutdown_connections 2024-04-25T11:21:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.015+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2662008476 wait complete. 2024-04-25T11:21:24.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 Processor -- start 2024-04-25T11:21:24.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- start start 2024-04-25T11:21:24.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f980c7c50 0x7f4f981f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f980c8e40 0x7f4f981510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9815cc80 con 0x7f4f980c4c90 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9815c9a0 con 0x7f4f980c8e40 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980c5270 con 0x7f4f980c7c50 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37482/0 (socket says 172.21.15.136:37482) 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 -- 172.21.15.136:0/2422355247 learned_addr learned my addr 172.21.15.136:0/2422355247 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f8ffff640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f980c7c50 0x7f4f981f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f96d9d640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f980c8e40 0x7f4f981510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 -- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f980c7c50 msgr2=0x7f4f981f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f980c7c50 0x7f4f981f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 -- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f980c8e40 msgr2=0x7f4f981510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f980c8e40 0x7f4f981510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 -- 172.21.15.136:0/2422355247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f98158190 con 0x7f4f980c4c90 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9659c640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f8001b860 tx=0x7f4f8001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800b5040 con 0x7f4f980c4c90 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f800bf070 con 0x7f4f980c4c90 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f98157d50 con 0x7f4f980c4c90 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800b0960 con 0x7f4f980c4c90 2024-04-25T11:21:24.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.019+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f98153da0 con 0x7f4f980c4c90 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9806a350 con 0x7f4f980c4c90 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4f800b0b00 con 0x7f4f980c4c90 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f8effd640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 0x7f4f7c071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4f8012c680 con 0x7f4f980c4c90 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f8ffff640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 0x7f4f7c071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.023+0000 7f4f8ffff640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 0x7f4f7c071960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f98151af0 tx=0x7f4f88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.027+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f800187b0 con 0x7f4f980c4c90 2024-04-25T11:21:24.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f980c63e0 con 0x7f4f980c4c90 2024-04-25T11:21:24.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f8effd640 1 -- 172.21.15.136:0/2422355247 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f4f800ff070 con 0x7f4f980c4c90 2024-04-25T11:21:24.177 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 msgr2=0x7f4f7c071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 0x7f4f7c071960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f98151af0 tx=0x7f4f88008040 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 msgr2=0x7f4f98156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4f8001b860 tx=0x7f4f8001b9f0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 shutdown_connections 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4f980c8e40 0x7f4f981510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4f7c06f450 0x7f4f7c071960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4f980c7c50 0x7f4f981f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 --2- 172.21.15.136:0/2422355247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f980c4c90 0x7f4f98156810 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 >> 172.21.15.136:0/2422355247 conn(0x7f4f980c9d50 msgr2=0x7f4f980cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 shutdown_connections 2024-04-25T11:21:24.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.179+0000 7f4f9cac9640 1 -- 172.21.15.136:0/2422355247 wait complete. 2024-04-25T11:21:24.196 INFO:tasks.mds_thrash:mds thrasher using random seed: 1714044084 2024-04-25T11:21:24.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:24.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 Processor -- start 2024-04-25T11:21:24.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 -- start start 2024-04-25T11:21:24.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 0x7f5848166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584815c6f0 con 0x7f584815ff60 2024-04-25T11:21:24.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f58480c6310 con 0x7f5848160370 2024-04-25T11:21:24.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584e831640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f584815c990 con 0x7f584815c0d0 2024-04-25T11:21:24.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584cdce640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 0x7f5848166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:43376/0 (socket says 172.21.15.136:43376) 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f584cdce640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 0x7f5848166a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:57224/0 (socket says 172.21.15.136:57224) 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 -- 172.21.15.136:0/3004991923 learned_addr learned my addr 172.21.15.136:0/3004991923 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f58477fe640 1 --1- 172.21.15.136:0/3004991923 >> v1:172.21.15.157:6789/0 conn(0x7f584815ff60 0x7f58480ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:51770/0 (socket says 172.21.15.136:51770) 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 -- 172.21.15.136:0/3004991923 >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 msgr2=0x7f5848166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 --2- 172.21.15.136:0/3004991923 >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 0x7f5848166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 -- 172.21.15.136:0/3004991923 >> v1:172.21.15.157:6789/0 conn(0x7f584815ff60 legacy=0x7f58480ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.347+0000 7f5847fff640 1 -- 172.21.15.136:0/3004991923 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584815cc30 con 0x7f584815c0d0 2024-04-25T11:21:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f5847fff640 1 --2- 172.21.15.136:0/3004991923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5838009870 tx=0x7f583801ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=833205600537ffde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f5846ffd640 1 -- 172.21.15.136:0/3004991923 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380b5070 con 0x7f584815c0d0 2024-04-25T11:21:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f5846ffd640 1 -- 172.21.15.136:0/3004991923 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58380a8a50 con 0x7f584815c0d0 2024-04-25T11:21:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f5846ffd640 1 -- 172.21.15.136:0/3004991923 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380a8e20 con 0x7f584815c0d0 2024-04-25T11:21:24.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 -- 172.21.15.136:0/3004991923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 msgr2=0x7f58480cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 --2- 172.21.15.136:0/3004991923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5838009870 tx=0x7f583801ed50 comp rx=0 tx=0).stop 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 -- 172.21.15.136:0/3004991923 shutdown_connections 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 --2- 172.21.15.136:0/3004991923 >> v2:172.21.15.157:3300/0 conn(0x7f5848160370 0x7f5848166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 --2- 172.21.15.136:0/3004991923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f584815c0d0 0x7f58480cb720 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 -- 172.21.15.136:0/3004991923 >> 172.21.15.136:0/3004991923 conn(0x7f584806f9d0 msgr2=0x7f584806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 -- 172.21.15.136:0/3004991923 shutdown_connections 2024-04-25T11:21:24.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.351+0000 7f584e831640 1 -- 172.21.15.136:0/3004991923 wait complete. 2024-04-25T11:21:24.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 Processor -- start 2024-04-25T11:21:24.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- start start 2024-04-25T11:21:24.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f584815ff60 0x7f58481e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5848160370 0x7f584814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f58480cbc60 con 0x7f584815c0d0 2024-04-25T11:21:24.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f584815fa40 con 0x7f584815ff60 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f584815cf10 con 0x7f5848160370 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37488/0 (socket says 172.21.15.136:37488) 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 -- 172.21.15.136:0/1847327575 learned_addr learned my addr 172.21.15.136:0/1847327575 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584cdce640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5848160370 0x7f584814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f58477fe640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f584815ff60 0x7f58481e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 -- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5848160370 msgr2=0x7f584814ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5848160370 0x7f584814ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 -- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f584815ff60 msgr2=0x7f58481e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f584815ff60 0x7f58481e70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 -- 172.21.15.136:0/1847327575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58480cd260 con 0x7f584815c0d0 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5847fff640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f58380b3040 tx=0x7f583801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380b5070 con 0x7f584815c0d0 2024-04-25T11:21:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58481e80b0 con 0x7f584815c0d0 2024-04-25T11:21:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58380b0d80 con 0x7f584815c0d0 2024-04-25T11:21:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58380bb650 con 0x7f584815c0d0 2024-04-25T11:21:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.355+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5848152bb0 con 0x7f584815c0d0 2024-04-25T11:21:24.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584806a350 con 0x7f584815c0d0 2024-04-25T11:21:24.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f58380c2060 con 0x7f584815c0d0 2024-04-25T11:21:24.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f5844ff9640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 0x7f5834071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f583812cb80 con 0x7f584815c0d0 2024-04-25T11:21:24.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f58477fe640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 0x7f5834071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.359+0000 7f58477fe640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 0x7f5834071940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5848150120 tx=0x7f583c007cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.363+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f58380188b0 con 0x7f584815c0d0 2024-04-25T11:21:24.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.515+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58480702f0 con 0x7f584815c0d0 2024-04-25T11:21:24.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f5844ff9640 1 -- 172.21.15.136:0/1847327575 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7f58380f9680 con 0x7f584815c0d0 2024-04-25T11:21:24.517 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:24.517 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:24.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 msgr2=0x7f5834071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 0x7f5834071940 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5848150120 tx=0x7f583c007cf0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 msgr2=0x7f58480ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f58380b3040 tx=0x7f583801baf0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 shutdown_connections 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f583406f430 0x7f5834071940 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5848160370 0x7f584814ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f584815ff60 0x7f58481e70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 --2- 172.21.15.136:0/1847327575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f584815c0d0 0x7f58480ccd20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 >> 172.21.15.136:0/1847327575 conn(0x7f584806f9d0 msgr2=0x7f58480ca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 shutdown_connections 2024-04-25T11:21:24.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.519+0000 7f584e831640 1 -- 172.21.15.136:0/1847327575 wait complete. 2024-04-25T11:21:24.537 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:21:24.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:21:24.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.687+0000 7fc070b50640 1 Processor -- start 2024-04-25T11:21:24.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.687+0000 7fc070b50640 1 -- start start 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc070b50640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 0x7fc06c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc070b50640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 0x7fc06c1639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc070b50640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c159900 con 0x7fc06c0c8e40 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc070b50640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc06c1596d0 con 0x7fc06c1615c0 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc070b50640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c159bc0 con 0x7fc06c0c7c50 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 0x7fc06c1639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 0x7fc06c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 0x7fc06c1639e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:57234/0 (socket says 172.21.15.136:57234) 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06ad9d640 1 -- 172.21.15.136:0/2820869580 learned_addr learned my addr 172.21.15.136:0/2820869580 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06a59c640 1 -- 172.21.15.136:0/2820869580 >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 msgr2=0x7fc06c1639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06a59c640 1 --2- 172.21.15.136:0/2820869580 >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 0x7fc06c1639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06a59c640 1 -- 172.21.15.136:0/2820869580 >> v1:172.21.15.157:6789/0 conn(0x7fc06c0c8e40 legacy=0x7fc06c0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.691+0000 7fc06a59c640 1 -- 172.21.15.136:0/2820869580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c15c3e0 con 0x7fc06c0c7c50 2024-04-25T11:21:24.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc06a59c640 1 --2- 172.21.15.136:0/2820869580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 0x7fc06c0c8900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00ba60 tx=0x7fc05c021360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e035aed247fdb6d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc069d9b640 1 -- 172.21.15.136:0/2820869580 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0b6070 con 0x7fc06c0c7c50 2024-04-25T11:21:24.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc069d9b640 1 -- 172.21.15.136:0/2820869580 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c0b1600 con 0x7fc06c0c7c50 2024-04-25T11:21:24.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc069d9b640 1 -- 172.21.15.136:0/2820869580 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0aae60 con 0x7fc06c0c7c50 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- 172.21.15.136:0/2820869580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 msgr2=0x7fc06c0c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 --2- 172.21.15.136:0/2820869580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 0x7fc06c0c8900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00ba60 tx=0x7fc05c021360 comp rx=0 tx=0).stop 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- 172.21.15.136:0/2820869580 shutdown_connections 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 --2- 172.21.15.136:0/2820869580 >> v2:172.21.15.157:3300/0 conn(0x7fc06c1615c0 0x7fc06c1639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 --2- 172.21.15.136:0/2820869580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c7c50 0x7fc06c0c8900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- 172.21.15.136:0/2820869580 >> 172.21.15.136:0/2820869580 conn(0x7fc06c0c9d50 msgr2=0x7fc06c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- 172.21.15.136:0/2820869580 shutdown_connections 2024-04-25T11:21:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- 172.21.15.136:0/2820869580 wait complete. 2024-04-25T11:21:24.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 Processor -- start 2024-04-25T11:21:24.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.695+0000 7fc070b50640 1 -- start start 2024-04-25T11:21:24.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c8e40 0x7fc06c15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc06c1615c0 0x7fc06c144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c15cc80 con 0x7fc06c0c7c50 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c15c9a0 con 0x7fc06c1615c0 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c0c5270 con 0x7fc06c0c8e40 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37010/0 (socket says 172.21.15.136:37010) 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 -- 172.21.15.136:0/2715982467 learned_addr learned my addr 172.21.15.136:0/2715982467 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc063fff640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c8e40 0x7fc06c15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06ad9d640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc06c1615c0 0x7fc06c144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 -- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c8e40 msgr2=0x7fc06c15b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c8e40 0x7fc06c15b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 -- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc06c1615c0 msgr2=0x7fc06c144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc06c1615c0 0x7fc06c144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 -- 172.21.15.136:0/2715982467 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c14c720 con 0x7fc06c0c7c50 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc06a59c640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0b4040 tx=0x7fc05c01e100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0c0070 con 0x7fc06c0c7c50 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05c0aae60 con 0x7fc06c0c7c50 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c159f40 con 0x7fc06c0c7c50 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c0b6040 con 0x7fc06c0c7c50 2024-04-25T11:21:24.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c163120 con 0x7fc06c0c7c50 2024-04-25T11:21:24.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc06c06a350 con 0x7fc06c0c7c50 2024-04-25T11:21:24.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc05c0c3070 con 0x7fc06c0c7c50 2024-04-25T11:21:24.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.699+0000 7fc062ffd640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 0x7fc04c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:24.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.703+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc05c12c860 con 0x7fc06c0c7c50 2024-04-25T11:21:24.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.703+0000 7fc063fff640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 0x7fc04c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:24.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.703+0000 7fc063fff640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 0x7fc04c071910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc06c072930 tx=0x7fc054009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:24.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.707+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc05c01aec0 con 0x7fc06c0c7c50 2024-04-25T11:21:24.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc06c0c63e0 con 0x7fc06c0c7c50 2024-04-25T11:21:24.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc062ffd640 1 -- 172.21.15.136:0/2715982467 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fc05c1013d0 con 0x7fc06c0c7c50 2024-04-25T11:21:24.865 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 msgr2=0x7fc04c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 0x7fc04c071910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc06c072930 tx=0x7fc054009040 comp rx=0 tx=0).stop 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 msgr2=0x7fc06c14ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0b4040 tx=0x7fc05c01e100 comp rx=0 tx=0).stop 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 shutdown_connections 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc06c1615c0 0x7fc06c144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc04c06f400 0x7fc04c071910 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc06c0c8e40 0x7fc06c15b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 --2- 172.21.15.136:0/2715982467 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc06c0c7c50 0x7fc06c14ad20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:24.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 >> 172.21.15.136:0/2715982467 conn(0x7fc06c0c9d50 msgr2=0x7fc06c0c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:24.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 shutdown_connections 2024-04-25T11:21:24.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:24.867+0000 7fc070b50640 1 -- 172.21.15.136:0/2715982467 wait complete. 2024-04-25T11:21:24.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:21:25.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.023+0000 7f678a038640 1 Processor -- start 2024-04-25T11:21:25.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.023+0000 7f678a038640 1 -- start start 2024-04-25T11:21:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6784156330 0x7f678415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840cdee0 0x7f67840ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67840b5490 0x7f67840b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f67841602f0 con 0x7f67840b5490 2024-04-25T11:21:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f678415d010 con 0x7f6784156330 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f678a038640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67840ccb80 con 0x7f67840cdee0 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67840b5490 0x7f67840b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67840b5490 0x7f67840b58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37018/0 (socket says 172.21.15.136:37018) 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 -- 172.21.15.136:0/2080824806 learned_addr learned my addr 172.21.15.136:0/2080824806 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6782ffd640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840cdee0 0x7f67840ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f67837fe640 1 --2- 172.21.15.136:0/2080824806 >> v2:172.21.15.157:3300/0 conn(0x7f6784156330 0x7f678415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 -- 172.21.15.136:0/2080824806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840cdee0 msgr2=0x7f67840ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840cdee0 0x7f67840ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 -- 172.21.15.136:0/2080824806 >> v2:172.21.15.157:3300/0 conn(0x7f6784156330 msgr2=0x7f678415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 --2- 172.21.15.136:0/2080824806 >> v2:172.21.15.157:3300/0 conn(0x7f6784156330 0x7f678415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.027+0000 7f6783fff640 1 -- 172.21.15.136:0/2080824806 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6784156740 con 0x7f67840b5490 2024-04-25T11:21:25.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.031+0000 7f6783fff640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67840b5490 0x7f67840b58a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f67840cf010 tx=0x7f677c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f98eb7d82bc268 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.031+0000 7f67827fc640 1 -- 172.21.15.136:0/2080824806 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c0b4070 con 0x7f67840b5490 2024-04-25T11:21:25.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.031+0000 7f67827fc640 1 -- 172.21.15.136:0/2080824806 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c0af400 con 0x7f67840b5490 2024-04-25T11:21:25.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.031+0000 7f67827fc640 1 -- 172.21.15.136:0/2080824806 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c0af7d0 con 0x7f67840b5490 2024-04-25T11:21:25.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- 172.21.15.136:0/2080824806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67840b5490 msgr2=0x7f67840b58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67840b5490 0x7f67840b58a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f67840cf010 tx=0x7f677c01f1e0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- 172.21.15.136:0/2080824806 shutdown_connections 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67840b5490 0x7f67840b58a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 --2- 172.21.15.136:0/2080824806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840cdee0 0x7f67840ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 --2- 172.21.15.136:0/2080824806 >> v2:172.21.15.157:3300/0 conn(0x7f6784156330 0x7f678415cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- 172.21.15.136:0/2080824806 >> 172.21.15.136:0/2080824806 conn(0x7f678406d6c0 msgr2=0x7f678406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- 172.21.15.136:0/2080824806 shutdown_connections 2024-04-25T11:21:25.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- 172.21.15.136:0/2080824806 wait complete. 2024-04-25T11:21:25.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 Processor -- start 2024-04-25T11:21:25.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.035+0000 7f678a038640 1 -- start start 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840b5490 0x7f678415d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67840cdee0 0x7f67840c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f678415fca0 con 0x7f6784156330 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f678415f9c0 con 0x7f67840cdee0 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f678415ff80 con 0x7f67840b5490 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37020/0 (socket says 172.21.15.136:37020) 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 -- 172.21.15.136:0/1363582711 learned_addr learned my addr 172.21.15.136:0/1363582711 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:25.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f67837fe640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840b5490 0x7f678415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6782ffd640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67840cdee0 0x7f67840c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 -- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840b5490 msgr2=0x7f678415d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840b5490 0x7f678415d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 -- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67840cdee0 msgr2=0x7f67840c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67840cdee0 0x7f67840c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 -- 172.21.15.136:0/1363582711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67841e6650 con 0x7f6784156330 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f6783fff640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f677c0b2040 tx=0x7f677c01bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c0b4040 con 0x7f6784156330 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c0c0070 con 0x7f6784156330 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67841e6930 con 0x7f6784156330 2024-04-25T11:21:25.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c009510 con 0x7f6784156330 2024-04-25T11:21:25.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67841e5780 con 0x7f6784156330 2024-04-25T11:21:25.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.039+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f678406a350 con 0x7f6784156330 2024-04-25T11:21:25.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.043+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f677c0c5460 con 0x7f6784156330 2024-04-25T11:21:25.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.043+0000 7f676bfff640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 0x7f6770071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.043+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f677c12e300 con 0x7f6784156330 2024-04-25T11:21:25.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.043+0000 7f67837fe640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 0x7f6770071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.043+0000 7f67837fe640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 0x7f6770071910 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f677400b930 tx=0x7f6774008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.047+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f677c018d40 con 0x7f6784156330 2024-04-25T11:21:25.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.195+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f67840cd9c0 con 0x7f6784156330 2024-04-25T11:21:25.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.195+0000 7f676bfff640 1 -- 172.21.15.136:0/1363582711 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f677c0faff0 con 0x7f6784156330 2024-04-25T11:21:25.194 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:25.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 msgr2=0x7f6770071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 0x7f6770071910 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f677400b930 tx=0x7f6774008040 comp rx=0 tx=0).stop 2024-04-25T11:21:25.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 msgr2=0x7f678414de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f677c0b2040 tx=0x7f677c01bf80 comp rx=0 tx=0).stop 2024-04-25T11:21:25.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 shutdown_connections 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6784156330 0x7f678414de70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67840cdee0 0x7f67840c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f677006f400 0x7f6770071910 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 --2- 172.21.15.136:0/1363582711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67840b5490 0x7f678415d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 >> 172.21.15.136:0/1363582711 conn(0x7f678406d6c0 msgr2=0x7f6784155020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 shutdown_connections 2024-04-25T11:21:25.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.199+0000 7f678a038640 1 -- 172.21.15.136:0/1363582711 wait complete. 2024-04-25T11:21:25.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:21:25.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.359+0000 7f3851425640 1 Processor -- start 2024-04-25T11:21:25.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.359+0000 7f3851425640 1 -- start start 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3851425640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3851425640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f384c1615c0 0x7f384c1639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3851425640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f384c159900 con 0x7f384c0c7c50 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3851425640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f384c1596d0 con 0x7f384c1615c0 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3851425640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f384c159bc0 con 0x7f384c0c4750 2024-04-25T11:21:25.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f384c1615c0 0x7f384c1639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384affd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f384c0c7c50 0x7f384c0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48094/0 (socket says 172.21.15.136:48094) 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384affd640 1 -- 172.21.15.136:0/3121769107 learned_addr learned my addr 172.21.15.136:0/3121769107 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384a7fc640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384b7fe640 1 -- 172.21.15.136:0/3121769107 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 msgr2=0x7f384c0c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384b7fe640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c0c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384b7fe640 1 -- 172.21.15.136:0/3121769107 >> v1:172.21.15.136:6789/0 conn(0x7f384c0c7c50 legacy=0x7f384c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384b7fe640 1 -- 172.21.15.136:0/3121769107 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c15c3e0 con 0x7f384c1615c0 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f3849ffb640 1 -- 172.21.15.136:0/3121769107 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761480226 0 0) 0x7f384c159900 con 0x7f384c0c7c50 2024-04-25T11:21:25.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.363+0000 7f384a7fc640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c0c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:25.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f384b7fe640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c1639e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25dcc272e985022 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3849ffb640 1 -- 172.21.15.136:0/3121769107 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38440b4070 con 0x7f384c1615c0 2024-04-25T11:21:25.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3849ffb640 1 -- 172.21.15.136:0/3121769107 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38440af600 con 0x7f384c1615c0 2024-04-25T11:21:25.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3849ffb640 1 -- 172.21.15.136:0/3121769107 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38440af9d0 con 0x7f384c1615c0 2024-04-25T11:21:25.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- 172.21.15.136:0/3121769107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 msgr2=0x7f384c1639e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c1639e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3844009d90 tx=0x7f384401f150 comp rx=0 tx=0).stop 2024-04-25T11:21:25.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- 172.21.15.136:0/3121769107 shutdown_connections 2024-04-25T11:21:25.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c1639e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 --2- 172.21.15.136:0/3121769107 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c0c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- 172.21.15.136:0/3121769107 >> 172.21.15.136:0/3121769107 conn(0x7f384c0c9d50 msgr2=0x7f384c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- 172.21.15.136:0/3121769107 shutdown_connections 2024-04-25T11:21:25.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- 172.21.15.136:0/3121769107 wait complete. 2024-04-25T11:21:25.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 Processor -- start 2024-04-25T11:21:25.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.367+0000 7f3851425640 1 -- start start 2024-04-25T11:21:25.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f384c0c7c50 0x7f384c143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f384c15cc80 con 0x7f384c1615c0 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f384c15c9a0 con 0x7f384c0c7c50 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f384c0c5270 con 0x7f384c0c4750 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37030/0 (socket says 172.21.15.136:37030) 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 -- 172.21.15.136:0/767044514 learned_addr learned my addr 172.21.15.136:0/767044514 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384affd640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384a7fc640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f384c0c7c50 0x7f384c143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 -- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 msgr2=0x7f384c1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 -- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f384c0c7c50 msgr2=0x7f384c143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f384c0c7c50 0x7f384c143df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 -- 172.21.15.136:0/767044514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c14ee50 con 0x7f384c1615c0 2024-04-25T11:21:25.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384b7fe640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f384401d800 tx=0x7f384401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38440b4040 con 0x7f384c1615c0 2024-04-25T11:21:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38440c2070 con 0x7f384c1615c0 2024-04-25T11:21:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384c15bb30 con 0x7f384c1615c0 2024-04-25T11:21:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3844009450 con 0x7f384c1615c0 2024-04-25T11:21:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384c1f5a90 con 0x7f384c1615c0 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384c06a350 con 0x7f384c1615c0 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f38440ba660 con 0x7f384c1615c0 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 0x7f3838071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f384412e940 con 0x7f384c1615c0 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.371+0000 7f384affd640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 0x7f3838071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.375+0000 7f384affd640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 0x7f3838071910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f383c000f70 tx=0x7f383c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.375+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3844018cb0 con 0x7f384c1615c0 2024-04-25T11:21:25.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.527+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f384c074bf0 con 0x7f384c1615c0 2024-04-25T11:21:25.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.527+0000 7f38337fe640 1 -- 172.21.15.136:0/767044514 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f38440fb6c0 con 0x7f384c1615c0 2024-04-25T11:21:25.526 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:21:25.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 msgr2=0x7f3838071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 0x7f3838071910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f383c000f70 tx=0x7f383c007c10 comp rx=0 tx=0).stop 2024-04-25T11:21:25.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 msgr2=0x7f384c145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f384401d800 tx=0x7f384401bef0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 shutdown_connections 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f384c1615c0 0x7f384c145980 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f384c0c7c50 0x7f384c143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f383806f400 0x7f3838071910 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 --2- 172.21.15.136:0/767044514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f384c0c4750 0x7f384c1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 >> 172.21.15.136:0/767044514 conn(0x7f384c0c9d50 msgr2=0x7f384c06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 shutdown_connections 2024-04-25T11:21:25.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.531+0000 7f3851425640 1 -- 172.21.15.136:0/767044514 wait complete. 2024-04-25T11:21:25.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:21:25.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 Processor -- start 2024-04-25T11:21:25.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 -- start start 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc9240c4750 0x7fc9240c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc924159900 con 0x7fc9240c7c50 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92415c3e0 con 0x7fc9241615c0 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc928deb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc9241596d0 con 0x7fc9240c4750 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc9240c7c50 0x7fc9240c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48102/0 (socket says 172.21.15.136:48102) 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 -- 172.21.15.136:0/202471346 learned_addr learned my addr 172.21.15.136:0/202471346 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc921d9b640 1 --2- 172.21.15.136:0/202471346 >> v2:172.21.15.157:3300/0 conn(0x7fc9240c4750 0x7fc9240c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703788799 0 0) 0x7fc924159900 con 0x7fc9240c7c50 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc90c003530 con 0x7fc9240c7c50 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1207531185 0 0) 0x7fc90c003530 con 0x7fc9240c7c50 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 >> v2:172.21.15.157:3300/0 conn(0x7fc9240c4750 msgr2=0x7fc9240c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 --2- 172.21.15.136:0/202471346 >> v2:172.21.15.157:3300/0 conn(0x7fc9240c4750 0x7fc9240c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 >> v1:172.21.15.157:6789/0 conn(0x7fc9241615c0 legacy=0x7fc9241639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92415c6c0 con 0x7fc9240c7c50 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fc914001680 con 0x7fc9240c7c50 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 >> v1:172.21.15.136:6789/0 conn(0x7fc9240c7c50 legacy=0x7fc9240c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 0x7fc90c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90c004810 0x7fc90c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0051d0 0x7fc90c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c00c130 con 0x7fc90c0039c0 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c00c2e0 con 0x7fc90c004810 2024-04-25T11:21:25.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc90c00c4e0 con 0x7fc90c0051d0 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 0x7fc90c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc922d9d640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0051d0 0x7fc90c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 -- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0051d0 msgr2=0x7fc90c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc921d9b640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90c004810 0x7fc90c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0051d0 0x7fc90c00b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 -- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90c004810 msgr2=0x7fc90c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc90c004810 0x7fc90c004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc92259c640 1 -- 172.21.15.136:0/202471346 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c003530 con 0x7fc90c0039c0 2024-04-25T11:21:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.687+0000 7fc922d9d640 1 -- 172.21.15.136:0/202471346 reap_dead start 2024-04-25T11:21:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc92259c640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 0x7fc90c003e30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c210 tx=0x7fc91401f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7514591341787a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914007d20 con 0x7fc90c0039c0 2024-04-25T11:21:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc914005ae0 con 0x7fc90c0039c0 2024-04-25T11:21:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc92159a640 1 -- 172.21.15.136:0/202471346 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc914007070 con 0x7fc90c0039c0 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 -- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 msgr2=0x7fc90c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 0x7fc90c003e30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc91400c210 tx=0x7fc91401f2f0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 -- 172.21.15.136:0/202471346 shutdown_connections 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 --2- 172.21.15.136:0/202471346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc90c0039c0 0x7fc90c003e30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 -- 172.21.15.136:0/202471346 >> 172.21.15.136:0/202471346 conn(0x7fc9240c9d50 msgr2=0x7fc9240ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 -- 172.21.15.136:0/202471346 shutdown_connections 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.691+0000 7fc928deb640 1 -- 172.21.15.136:0/202471346 wait complete. 2024-04-25T11:21:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 Processor -- start 2024-04-25T11:21:25.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- start start 2024-04-25T11:21:25.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0039c0 0x7fc9241f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9240cbd80 0x7fc92414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc92415cc80 con 0x7fc924145a10 2024-04-25T11:21:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc924159bc0 con 0x7fc9240cbd80 2024-04-25T11:21:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9240c5270 con 0x7fc90c0039c0 2024-04-25T11:21:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37062/0 (socket says 172.21.15.136:37062) 2024-04-25T11:21:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 -- 172.21.15.136:0/730906374 learned_addr learned my addr 172.21.15.136:0/730906374 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc92259c640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0039c0 0x7fc9241f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 -- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0039c0 msgr2=0x7fc9241f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0039c0 0x7fc9241f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 -- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9240cbd80 msgr2=0x7fc92414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc921d9b640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9240cbd80 0x7fc92414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9240cbd80 0x7fc92414e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 -- 172.21.15.136:0/730906374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9241f46a0 con 0x7fc924145a10 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc922d9d640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc9240cad70 tx=0x7fc91c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c018070 con 0x7fc924145a10 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c003b80 con 0x7fc924145a10 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9240cc4d0 con 0x7fc924145a10 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c005070 con 0x7fc924145a10 2024-04-25T11:21:25.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9240cb2a0 con 0x7fc924145a10 2024-04-25T11:21:25.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.695+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92406a350 con 0x7fc924145a10 2024-04-25T11:21:25.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.699+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc91c005210 con 0x7fc924145a10 2024-04-25T11:21:25.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.699+0000 7fc912ffd640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 0x7fc904071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:25.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.699+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc91c08ec10 con 0x7fc924145a10 2024-04-25T11:21:25.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.699+0000 7fc92259c640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 0x7fc904071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:25.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.699+0000 7fc92259c640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 0x7fc904071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9140af000 tx=0x7fc9140aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:25.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.703+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc91c05b820 con 0x7fc924145a10 2024-04-25T11:21:25.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.851+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc9240ccbd0 con 0x7fc924145a10 2024-04-25T11:21:25.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.851+0000 7fc912ffd640 1 -- 172.21.15.136:0/730906374 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc91c05b120 con 0x7fc924145a10 2024-04-25T11:21:25.850 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 msgr2=0x7fc904071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 0x7fc904071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9140af000 tx=0x7fc9140aa000 comp rx=0 tx=0).stop 2024-04-25T11:21:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 msgr2=0x7fc924145e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc9240cad70 tx=0x7fc91c00dc00 comp rx=0 tx=0).stop 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 shutdown_connections 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc924145a10 0x7fc924145e20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9240cbd80 0x7fc92414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc90406f430 0x7fc904071940 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 --2- 172.21.15.136:0/730906374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc90c0039c0 0x7fc9241f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 >> 172.21.15.136:0/730906374 conn(0x7fc9240c9d50 msgr2=0x7fc924147cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 shutdown_connections 2024-04-25T11:21:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:25.855+0000 7fc928deb640 1 -- 172.21.15.136:0/730906374 wait complete. 2024-04-25T11:21:25.870 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-25T11:21:25.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:21:26.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 Processor -- start 2024-04-25T11:21:26.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 -- start start 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efea80b5060 0x7efea80c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 0x7efea80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea815d750 con 0x7efea815a430 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efea815da30 con 0x7efea80b5060 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeae4cf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efea815dd10 con 0x7efea80ca290 2024-04-25T11:21:26.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeaca6c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7efea815a430 0x7efea815a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48104/0 (socket says 172.21.15.136:48104) 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efeaca6c640 1 -- 172.21.15.136:0/730790502 learned_addr learned my addr 172.21.15.136:0/730790502 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea77fe640 1 --2- 172.21.15.136:0/730790502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 0x7efea80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea7fff640 1 --2- 172.21.15.136:0/730790502 >> v2:172.21.15.157:3300/0 conn(0x7efea80b5060 0x7efea80c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4080022383 0 0) 0x7efea815d750 con 0x7efea815a430 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe94003530 con 0x7efea815a430 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4040362591 0 0) 0x7efe94003530 con 0x7efea815a430 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea77fe640 1 -- 172.21.15.136:0/730790502 >> v2:172.21.15.157:3300/0 conn(0x7efea80b5060 msgr2=0x7efea80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea77fe640 1 --2- 172.21.15.136:0/730790502 >> v2:172.21.15.157:3300/0 conn(0x7efea80b5060 0x7efea80c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea77fe640 1 -- 172.21.15.136:0/730790502 >> v1:172.21.15.136:6789/0 conn(0x7efea815a430 legacy=0x7efea815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.015+0000 7efea77fe640 1 -- 172.21.15.136:0/730790502 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea815dff0 con 0x7efea80ca290 2024-04-25T11:21:26.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efea77fe640 1 --2- 172.21.15.136:0/730790502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 0x7efea80ca6a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe900097c0 tx=0x7efe9001ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780e9a7fad49ec65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe900b5070 con 0x7efea80ca290 2024-04-25T11:21:26.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe900a85f0 con 0x7efea80ca290 2024-04-25T11:21:26.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efea6ffd640 1 -- 172.21.15.136:0/730790502 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0001a00 con 0x7efea80ca290 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 -- 172.21.15.136:0/730790502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 msgr2=0x7efea80ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 --2- 172.21.15.136:0/730790502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 0x7efea80ca6a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe900097c0 tx=0x7efe9001ebb0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 -- 172.21.15.136:0/730790502 shutdown_connections 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 --2- 172.21.15.136:0/730790502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea80ca290 0x7efea80ca6a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 --2- 172.21.15.136:0/730790502 >> v2:172.21.15.157:3300/0 conn(0x7efea80b5060 0x7efea80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 -- 172.21.15.136:0/730790502 >> 172.21.15.136:0/730790502 conn(0x7efea8070f70 msgr2=0x7efea8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 -- 172.21.15.136:0/730790502 shutdown_connections 2024-04-25T11:21:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.019+0000 7efeae4cf640 1 -- 172.21.15.136:0/730790502 wait complete. 2024-04-25T11:21:26.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 Processor -- start 2024-04-25T11:21:26.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- start start 2024-04-25T11:21:26.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efea80ca290 0x7efea80b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea815a430 0x7efea80b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efea815d3f0 con 0x7efea80b5060 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efea8159f10 con 0x7efea80ca290 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efea80c4a60 con 0x7efea815a430 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37074/0 (socket says 172.21.15.136:37074) 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 -- 172.21.15.136:0/899475687 learned_addr learned my addr 172.21.15.136:0/899475687 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeaca6c640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea815a430 0x7efea80b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea77fe640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efea80ca290 0x7efea80b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 -- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea815a430 msgr2=0x7efea80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea815a430 0x7efea80b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 -- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efea80ca290 msgr2=0x7efea80b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efea80ca290 0x7efea80b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 -- 172.21.15.136:0/899475687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea81f3350 con 0x7efea80b5060 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea7fff640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e870 tx=0x7efe9800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98017070 con 0x7efea80b5060 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9800c8e0 con 0x7efea80b5060 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea81f3630 con 0x7efea80b5060 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9800cc90 con 0x7efea80b5060 2024-04-25T11:21:26.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.023+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea81f4110 con 0x7efea80b5060 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea806a350 con 0x7efea80b5060 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efe9801a070 con 0x7efea80b5060 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efea4ff9640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 0x7efe88071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efe9808fc90 con 0x7efea80b5060 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efea77fe640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 0x7efe88071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.027+0000 7efea77fe640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 0x7efe88071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe900b2000 tx=0x7efe900aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.031+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efe9805c8a0 con 0x7efea80b5060 2024-04-25T11:21:26.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.223+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efea8150f90 con 0x7efea80b5060 2024-04-25T11:21:26.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.223+0000 7efea4ff9640 1 -- 172.21.15.136:0/899475687 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7efe9805c1a0 con 0x7efea80b5060 2024-04-25T11:21:26.223 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:26.223 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:21:26.223 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:21:26.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 msgr2=0x7efe88071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 0x7efe88071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe900b2000 tx=0x7efe900aa070 comp rx=0 tx=0).stop 2024-04-25T11:21:26.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 msgr2=0x7efea8157f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe9800e870 tx=0x7efe9800ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 shutdown_connections 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efe8806f400 0x7efe88071910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efea815a430 0x7efea80b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efea80ca290 0x7efea80b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 --2- 172.21.15.136:0/899475687 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efea80b5060 0x7efea8157f90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 >> 172.21.15.136:0/899475687 conn(0x7efea8070f70 msgr2=0x7efea8075ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 shutdown_connections 2024-04-25T11:21:26.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.227+0000 7efeae4cf640 1 -- 172.21.15.136:0/899475687 wait complete. 2024-04-25T11:21:26.243 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-25T11:21:26.243 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-25T11:21:26.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:21:26.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.391+0000 7f5e05aeb640 1 Processor -- start 2024-04-25T11:21:26.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.391+0000 7f5e05aeb640 1 -- start start 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5e05aeb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5e00070870 0x7f5e000c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5e05aeb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5e00070c80 0x7f5e000c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5e05aeb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e00143580 con 0x7f5e00143170 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5e05aeb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5e0015dd20 con 0x7f5e00070c80 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5e05aeb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5e0015e000 con 0x7f5e00070870 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5e00070c80 0x7f5e000c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5e00070c80 0x7f5e000c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37082/0 (socket says 172.21.15.136:37082) 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 -- 172.21.15.136:0/3944969261 learned_addr learned my addr 172.21.15.136:0/3944969261 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfeffd640 1 --2- 172.21.15.136:0/3944969261 >> v2:172.21.15.157:3300/0 conn(0x7f5e00070870 0x7f5e000c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 -- 172.21.15.136:0/3944969261 >> v2:172.21.15.157:3300/0 conn(0x7f5e00070870 msgr2=0x7f5e000c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 --2- 172.21.15.136:0/3944969261 >> v2:172.21.15.157:3300/0 conn(0x7f5e00070870 0x7f5e000c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 -- 172.21.15.136:0/3944969261 >> v1:172.21.15.157:6789/0 conn(0x7f5e00143170 legacy=0x7f5e0015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:26.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.395+0000 7f5dfffff640 1 -- 172.21.15.136:0/3944969261 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0006fd90 con 0x7f5e00070c80 2024-04-25T11:21:26.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5dfffff640 1 --2- 172.21.15.136:0/3944969261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c6ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5e00157a80 tx=0x7f5df801efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7deb23abbac9e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5dfe7fc640 1 -- 172.21.15.136:0/3944969261 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df80a8bd0 con 0x7f5e00070c80 2024-04-25T11:21:26.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5dfe7fc640 1 -- 172.21.15.136:0/3944969261 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df80a8d70 con 0x7f5e00070c80 2024-04-25T11:21:26.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5dfe7fc640 1 -- 172.21.15.136:0/3944969261 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df80b0610 con 0x7f5e00070c80 2024-04-25T11:21:26.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- 172.21.15.136:0/3944969261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 msgr2=0x7f5e000c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/3944969261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c6ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5e00157a80 tx=0x7f5df801efc0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- 172.21.15.136:0/3944969261 shutdown_connections 2024-04-25T11:21:26.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/3944969261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c6ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/3944969261 >> v2:172.21.15.157:3300/0 conn(0x7f5e00070870 0x7f5e000c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- 172.21.15.136:0/3944969261 >> 172.21.15.136:0/3944969261 conn(0x7f5e000c9d50 msgr2=0x7f5e000ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:26.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- 172.21.15.136:0/3944969261 shutdown_connections 2024-04-25T11:21:26.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- 172.21.15.136:0/3944969261 wait complete. 2024-04-25T11:21:26.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 Processor -- start 2024-04-25T11:21:26.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.399+0000 7f5e05aeb640 1 -- start start 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 0x7f5e00153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e00143170 0x7f5e000c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0015e2e0 con 0x7f5e00070c80 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e00070350 con 0x7f5e00143170 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e000712a0 con 0x7f5e00070870 2024-04-25T11:21:26.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37086/0 (socket says 172.21.15.136:37086) 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 -- 172.21.15.136:0/2866076706 learned_addr learned my addr 172.21.15.136:0/2866076706 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dff7fe640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 0x7f5e00153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfffff640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e00143170 0x7f5e000c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 -- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 msgr2=0x7f5e00153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 0x7f5e00153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 -- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e00143170 msgr2=0x7f5e000c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e00143170 0x7f5e000c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 -- 172.21.15.136:0/2866076706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e000cc580 con 0x7f5e00070c80 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dff7fe640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 0x7f5e00153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dfeffd640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c5e0 tx=0x7f5dec00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec015070 con 0x7f5e00070c80 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dec004220 con 0x7f5e00070c80 2024-04-25T11:21:26.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e000cc860 con 0x7f5e00070c80 2024-04-25T11:21:26.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec003620 con 0x7f5e00070c80 2024-04-25T11:21:26.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e000cb1c0 con 0x7f5e00070c80 2024-04-25T11:21:26.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e0006a350 con 0x7f5e00070c80 2024-04-25T11:21:26.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5dec0037e0 con 0x7f5e00070c80 2024-04-25T11:21:26.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 0x7f5de8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5dec08f3c0 con 0x7f5e00070c80 2024-04-25T11:21:26.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.403+0000 7f5dff7fe640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 0x7f5de8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.407+0000 7f5dff7fe640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 0x7f5de8071910 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5df000a850 tx=0x7f5df0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.411+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5dec05bfd0 con 0x7f5e00070c80 2024-04-25T11:21:26.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.567+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e00140250 con 0x7f5de806f400 2024-04-25T11:21:26.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.595+0000 7f5df7fff640 1 -- 172.21.15.136:0/2866076706 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332671 (secure 0 0 0) 0x7f5e00140250 con 0x7f5de806f400 2024-04-25T11:21:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:26.598 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:21:26.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 msgr2=0x7f5de8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 0x7f5de8071910 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5df000a850 tx=0x7f5df0008040 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 msgr2=0x7f5e000c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c5e0 tx=0x7f5dec00caa0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 shutdown_connections 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e00143170 0x7f5e000c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e00070c80 0x7f5e000c58d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5de806f400 0x7f5de8071910 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 --2- 172.21.15.136:0/2866076706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e00070870 0x7f5e00153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 >> 172.21.15.136:0/2866076706 conn(0x7f5e000c9d50 msgr2=0x7f5e00073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 shutdown_connections 2024-04-25T11:21:26.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.599+0000 7f5e05aeb640 1 -- 172.21.15.136:0/2866076706 wait complete. 2024-04-25T11:21:26.617 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-04-25T11:21:24.779333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601955,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":315,"num_write_kb":1700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236360,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738360,"statfs":{"total":773094113280,"available":772852080640,"internally_reserved":0,"allocated":3678208,"data_stored":2116152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2729,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005794"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.994195+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:21:11.994195+0000","last_peered":"2024-04-25T11:21:11.994195+0000","last_clean":"2024-04-25T11:21:11.994195+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:21:11.994195+0000","last_undegraded":"2024-04-25T11:21:11.994195+0000","last_fullsized":"2024-04-25T11:21:11.994195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931730+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:59.931730+0000","last_peered":"2024-04-25T11:20:59.931730+0000","last_clean":"2024-04-25T11:20:59.931730+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:59.931730+0000","last_undegraded":"2024-04-25T11:20:59.931730+0000","last_fullsized":"2024-04-25T11:20:59.931730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928414+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:59.928414+0000","last_peered":"2024-04-25T11:20:59.928414+0000","last_clean":"2024-04-25T11:20:59.928414+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:59.928414+0000","last_undegraded":"2024-04-25T11:20:59.928414+0000","last_fullsized":"2024-04-25T11:20:59.928414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932201+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:59.932201+0000","last_peered":"2024-04-25T11:20:59.932201+0000","last_clean":"2024-04-25T11:20:59.932201+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:59.932201+0000","last_undegraded":"2024-04-25T11:20:59.932201+0000","last_fullsized":"2024-04-25T11:20:59.932201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940417+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:59.940417+0000","last_peered":"2024-04-25T11:20:59.940417+0000","last_clean":"2024-04-25T11:20:59.940417+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:59.940417+0000","last_undegraded":"2024-04-25T11:20:59.940417+0000","last_fullsized":"2024-04-25T11:20:59.940417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068980+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:21:00.068980+0000","last_peered":"2024-04-25T11:21:00.068980+0000","last_clean":"2024-04-25T11:21:00.068980+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:21:00.068980+0000","last_undegraded":"2024-04-25T11:21:00.068980+0000","last_fullsized":"2024-04-25T11:21:00.068980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.003480+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:21:12.003480+0000","last_peered":"2024-04-25T11:21:12.003480+0000","last_clean":"2024-04-25T11:21:12.003480+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:12.003480+0000","last_undegraded":"2024-04-25T11:21:12.003480+0000","last_fullsized":"2024-04-25T11:21:12.003480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075805+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:21:00.075805+0000","last_peered":"2024-04-25T11:21:00.075805+0000","last_clean":"2024-04-25T11:21:00.075805+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:21:00.075805+0000","last_undegraded":"2024-04-25T11:21:00.075805+0000","last_fullsized":"2024-04-25T11:21:00.075805+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069307+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:21:00.069307+0000","last_peered":"2024-04-25T11:21:00.069307+0000","last_clean":"2024-04-25T11:21:00.069307+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:21:00.069307+0000","last_undegraded":"2024-04-25T11:21:00.069307+0000","last_fullsized":"2024-04-25T11:21:00.069307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002064+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:21:12.002064+0000","last_peered":"2024-04-25T11:21:12.002064+0000","last_clean":"2024-04-25T11:21:12.002064+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:21:12.002064+0000","last_undegraded":"2024-04-25T11:21:12.002064+0000","last_fullsized":"2024-04-25T11:21:12.002064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075553+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:21:00.075553+0000","last_peered":"2024-04-25T11:21:00.075553+0000","last_clean":"2024-04-25T11:21:00.075553+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:21:00.075553+0000","last_undegraded":"2024-04-25T11:21:00.075553+0000","last_fullsized":"2024-04-25T11:21:00.075553+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069045+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:21:00.069045+0000","last_peered":"2024-04-25T11:21:00.069045+0000","last_clean":"2024-04-25T11:21:00.069045+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:21:00.069045+0000","last_undegraded":"2024-04-25T11:21:00.069045+0000","last_fullsized":"2024-04-25T11:21:00.069045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924874+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:59.924874+0000","last_peered":"2024-04-25T11:20:59.924874+0000","last_clean":"2024-04-25T11:20:59.924874+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:59.924874+0000","last_undegraded":"2024-04-25T11:20:59.924874+0000","last_fullsized":"2024-04-25T11:20:59.924874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111184+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:21:00.111184+0000","last_peered":"2024-04-25T11:21:00.111184+0000","last_clean":"2024-04-25T11:21:00.111184+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:21:00.111184+0000","last_undegraded":"2024-04-25T11:21:00.111184+0000","last_fullsized":"2024-04-25T11:21:00.111184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070128+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:21:00.070128+0000","last_peered":"2024-04-25T11:21:00.070128+0000","last_clean":"2024-04-25T11:21:00.070128+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:21:00.070128+0000","last_undegraded":"2024-04-25T11:21:00.070128+0000","last_fullsized":"2024-04-25T11:21:00.070128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930588+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:59.930588+0000","last_peered":"2024-04-25T11:20:59.930588+0000","last_clean":"2024-04-25T11:20:59.930588+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:59.930588+0000","last_undegraded":"2024-04-25T11:20:59.930588+0000","last_fullsized":"2024-04-25T11:20:59.930588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959714+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:21:05.959714+0000","last_peered":"2024-04-25T11:21:05.959714+0000","last_clean":"2024-04-25T11:21:05.959714+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:21:05.959714+0000","last_undegraded":"2024-04-25T11:21:05.959714+0000","last_fullsized":"2024-04-25T11:21:05.959714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069432+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:21:00.069432+0000","last_peered":"2024-04-25T11:21:00.069432+0000","last_clean":"2024-04-25T11:21:00.069432+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:21:00.069432+0000","last_undegraded":"2024-04-25T11:21:00.069432+0000","last_fullsized":"2024-04-25T11:21:00.069432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926697+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:59.926697+0000","last_peered":"2024-04-25T11:20:59.926697+0000","last_clean":"2024-04-25T11:20:59.926697+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:59.926697+0000","last_undegraded":"2024-04-25T11:20:59.926697+0000","last_fullsized":"2024-04-25T11:20:59.926697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.991983+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:21:11.991983+0000","last_peered":"2024-04-25T11:21:11.991983+0000","last_clean":"2024-04-25T11:21:11.991983+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:21:11.991983+0000","last_undegraded":"2024-04-25T11:21:11.991983+0000","last_fullsized":"2024-04-25T11:21:11.991983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.114813+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:21:00.114813+0000","last_peered":"2024-04-25T11:21:00.114813+0000","last_clean":"2024-04-25T11:21:00.114813+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:21:00.114813+0000","last_undegraded":"2024-04-25T11:21:00.114813+0000","last_fullsized":"2024-04-25T11:21:00.114813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067738+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:21:00.067738+0000","last_peered":"2024-04-25T11:21:00.067738+0000","last_clean":"2024-04-25T11:21:00.067738+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:21:00.067738+0000","last_undegraded":"2024-04-25T11:21:00.067738+0000","last_fullsized":"2024-04-25T11:21:00.067738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110139+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:21:00.110139+0000","last_peered":"2024-04-25T11:21:00.110139+0000","last_clean":"2024-04-25T11:21:00.110139+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:21:00.110139+0000","last_undegraded":"2024-04-25T11:21:00.110139+0000","last_fullsized":"2024-04-25T11:21:00.110139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070453+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:21:00.070453+0000","last_peered":"2024-04-25T11:21:00.070453+0000","last_clean":"2024-04-25T11:21:00.070453+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:21:00.070453+0000","last_undegraded":"2024-04-25T11:21:00.070453+0000","last_fullsized":"2024-04-25T11:21:00.070453+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940092+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:59.940092+0000","last_peered":"2024-04-25T11:20:59.940092+0000","last_clean":"2024-04-25T11:20:59.940092+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:59.940092+0000","last_undegraded":"2024-04-25T11:20:59.940092+0000","last_fullsized":"2024-04-25T11:20:59.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002572+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:21:12.002572+0000","last_peered":"2024-04-25T11:21:12.002572+0000","last_clean":"2024-04-25T11:21:12.002572+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:21:12.002572+0000","last_undegraded":"2024-04-25T11:21:12.002572+0000","last_fullsized":"2024-04-25T11:21:12.002572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069767+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:21:00.069767+0000","last_peered":"2024-04-25T11:21:00.069767+0000","last_clean":"2024-04-25T11:21:00.069767+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:21:00.069767+0000","last_undegraded":"2024-04-25T11:21:00.069767+0000","last_fullsized":"2024-04-25T11:21:00.069767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932437+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:59.932437+0000","last_peered":"2024-04-25T11:20:59.932437+0000","last_clean":"2024-04-25T11:20:59.932437+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:59.932437+0000","last_undegraded":"2024-04-25T11:20:59.932437+0000","last_fullsized":"2024-04-25T11:20:59.932437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962037+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:21:05.962037+0000","last_peered":"2024-04-25T11:21:05.962037+0000","last_clean":"2024-04-25T11:21:05.962037+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:21:05.962037+0000","last_undegraded":"2024-04-25T11:21:05.962037+0000","last_fullsized":"2024-04-25T11:21:05.962037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925818+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:59.925818+0000","last_peered":"2024-04-25T11:20:59.925818+0000","last_clean":"2024-04-25T11:20:59.925818+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:59.925818+0000","last_undegraded":"2024-04-25T11:20:59.925818+0000","last_fullsized":"2024-04-25T11:20:59.925818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936496+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:59.936496+0000","last_peered":"2024-04-25T11:20:59.936496+0000","last_clean":"2024-04-25T11:20:59.936496+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:59.936496+0000","last_undegraded":"2024-04-25T11:20:59.936496+0000","last_fullsized":"2024-04-25T11:20:59.936496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962135+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:21:05.962135+0000","last_peered":"2024-04-25T11:21:05.962135+0000","last_clean":"2024-04-25T11:21:05.962135+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:05.962135+0000","last_undegraded":"2024-04-25T11:21:05.962135+0000","last_fullsized":"2024-04-25T11:21:05.962135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935995+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:59.935995+0000","last_peered":"2024-04-25T11:20:59.935995+0000","last_clean":"2024-04-25T11:20:59.935995+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:59.935995+0000","last_undegraded":"2024-04-25T11:20:59.935995+0000","last_fullsized":"2024-04-25T11:20:59.935995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474622+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474622+0000","last_peered":"2024-04-25T11:21:00.474622+0000","last_clean":"2024-04-25T11:21:00.474622+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:21:00.474622+0000","last_undegraded":"2024-04-25T11:21:00.474622+0000","last_fullsized":"2024-04-25T11:21:00.474622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936918+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:59.936918+0000","last_peered":"2024-04-25T11:20:59.936918+0000","last_clean":"2024-04-25T11:20:59.936918+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:59.936918+0000","last_undegraded":"2024-04-25T11:20:59.936918+0000","last_fullsized":"2024-04-25T11:20:59.936918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936370+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:59.936370+0000","last_peered":"2024-04-25T11:20:59.936370+0000","last_clean":"2024-04-25T11:20:59.936370+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:59.936370+0000","last_undegraded":"2024-04-25T11:20:59.936370+0000","last_fullsized":"2024-04-25T11:20:59.936370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068094+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:21:00.068094+0000","last_peered":"2024-04-25T11:21:00.068094+0000","last_clean":"2024-04-25T11:21:00.068094+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:21:00.068094+0000","last_undegraded":"2024-04-25T11:21:00.068094+0000","last_fullsized":"2024-04-25T11:21:00.068094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.999818+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:21:11.999818+0000","last_peered":"2024-04-25T11:21:11.999818+0000","last_clean":"2024-04-25T11:21:11.999818+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:21:11.999818+0000","last_undegraded":"2024-04-25T11:21:11.999818+0000","last_fullsized":"2024-04-25T11:21:11.999818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:59.936416+0000","last_peered":"2024-04-25T11:20:59.936416+0000","last_clean":"2024-04-25T11:20:59.936416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:59.936416+0000","last_undegraded":"2024-04-25T11:20:59.936416+0000","last_fullsized":"2024-04-25T11:20:59.936416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.934845+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:59.934845+0000","last_peered":"2024-04-25T11:20:59.934845+0000","last_clean":"2024-04-25T11:20:59.934845+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:59.934845+0000","last_undegraded":"2024-04-25T11:20:59.934845+0000","last_fullsized":"2024-04-25T11:20:59.934845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992803+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:21:11.992803+0000","last_peered":"2024-04-25T11:21:11.992803+0000","last_clean":"2024-04-25T11:21:11.992803+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:21:11.992803+0000","last_undegraded":"2024-04-25T11:21:11.992803+0000","last_fullsized":"2024-04-25T11:21:11.992803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939030+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:59.939030+0000","last_peered":"2024-04-25T11:20:59.939030+0000","last_clean":"2024-04-25T11:20:59.939030+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:59.939030+0000","last_undegraded":"2024-04-25T11:20:59.939030+0000","last_fullsized":"2024-04-25T11:20:59.939030+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933545+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:59.933545+0000","last_peered":"2024-04-25T11:20:59.933545+0000","last_clean":"2024-04-25T11:20:59.933545+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:59.933545+0000","last_undegraded":"2024-04-25T11:20:59.933545+0000","last_fullsized":"2024-04-25T11:20:59.933545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.917110+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:21:00.917110+0000","last_peered":"2024-04-25T11:21:00.917110+0000","last_clean":"2024-04-25T11:21:00.917110+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:21:00.917110+0000","last_undegraded":"2024-04-25T11:21:00.917110+0000","last_fullsized":"2024-04-25T11:21:00.917110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939772+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:59.939772+0000","last_peered":"2024-04-25T11:20:59.939772+0000","last_clean":"2024-04-25T11:20:59.939772+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:59.939772+0000","last_undegraded":"2024-04-25T11:20:59.939772+0000","last_fullsized":"2024-04-25T11:20:59.939772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929859+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:59.929859+0000","last_peered":"2024-04-25T11:20:59.929859+0000","last_clean":"2024-04-25T11:20:59.929859+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:59.929859+0000","last_undegraded":"2024-04-25T11:20:59.929859+0000","last_fullsized":"2024-04-25T11:20:59.929859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960397+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:21:05.960397+0000","last_peered":"2024-04-25T11:21:05.960397+0000","last_clean":"2024-04-25T11:21:05.960397+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:21:05.960397+0000","last_undegraded":"2024-04-25T11:21:05.960397+0000","last_fullsized":"2024-04-25T11:21:05.960397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":122,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110312+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:21:00.110312+0000","last_peered":"2024-04-25T11:21:00.110312+0000","last_clean":"2024-04-25T11:21:00.110312+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:21:00.110312+0000","last_undegraded":"2024-04-25T11:21:00.110312+0000","last_fullsized":"2024-04-25T11:21:00.110312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924917+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:59.924917+0000","last_peered":"2024-04-25T11:20:59.924917+0000","last_clean":"2024-04-25T11:20:59.924917+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:59.924917+0000","last_undegraded":"2024-04-25T11:20:59.924917+0000","last_fullsized":"2024-04-25T11:20:59.924917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111023+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:21:00.111023+0000","last_peered":"2024-04-25T11:21:00.111023+0000","last_clean":"2024-04-25T11:21:00.111023+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:21:00.111023+0000","last_undegraded":"2024-04-25T11:21:00.111023+0000","last_fullsized":"2024-04-25T11:21:00.111023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961234+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:21:05.961234+0000","last_peered":"2024-04-25T11:21:05.961234+0000","last_clean":"2024-04-25T11:21:05.961234+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:21:05.961234+0000","last_undegraded":"2024-04-25T11:21:05.961234+0000","last_fullsized":"2024-04-25T11:21:05.961234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924269+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:59.924269+0000","last_peered":"2024-04-25T11:20:59.924269+0000","last_clean":"2024-04-25T11:20:59.924269+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:59.924269+0000","last_undegraded":"2024-04-25T11:20:59.924269+0000","last_fullsized":"2024-04-25T11:20:59.924269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474615+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:21:00.474615+0000","last_peered":"2024-04-25T11:21:00.474615+0000","last_clean":"2024-04-25T11:21:00.474615+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:21:00.474615+0000","last_undegraded":"2024-04-25T11:21:00.474615+0000","last_fullsized":"2024-04-25T11:21:00.474615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961055+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:21:05.961055+0000","last_peered":"2024-04-25T11:21:05.961055+0000","last_clean":"2024-04-25T11:21:05.961055+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:21:05.961055+0000","last_undegraded":"2024-04-25T11:21:05.961055+0000","last_fullsized":"2024-04-25T11:21:05.961055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.479066+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:21:00.479066+0000","last_peered":"2024-04-25T11:21:00.479066+0000","last_clean":"2024-04-25T11:21:00.479066+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:21:00.479066+0000","last_undegraded":"2024-04-25T11:21:00.479066+0000","last_fullsized":"2024-04-25T11:21:00.479066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111834+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:21:00.111834+0000","last_peered":"2024-04-25T11:21:00.111834+0000","last_clean":"2024-04-25T11:21:00.111834+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:21:00.111834+0000","last_undegraded":"2024-04-25T11:21:00.111834+0000","last_fullsized":"2024-04-25T11:21:00.111834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992924+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:21:11.992924+0000","last_peered":"2024-04-25T11:21:11.992924+0000","last_clean":"2024-04-25T11:21:11.992924+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:21:11.992924+0000","last_undegraded":"2024-04-25T11:21:11.992924+0000","last_fullsized":"2024-04-25T11:21:11.992924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474787+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:21:00.474787+0000","last_peered":"2024-04-25T11:21:00.474787+0000","last_clean":"2024-04-25T11:21:00.474787+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:21:00.474787+0000","last_undegraded":"2024-04-25T11:21:00.474787+0000","last_fullsized":"2024-04-25T11:21:00.474787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111717+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:21:00.111717+0000","last_peered":"2024-04-25T11:21:00.111717+0000","last_clean":"2024-04-25T11:21:00.111717+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:21:00.111717+0000","last_undegraded":"2024-04-25T11:21:00.111717+0000","last_fullsized":"2024-04-25T11:21:00.111717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070606+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:21:00.070606+0000","last_peered":"2024-04-25T11:21:00.070606+0000","last_clean":"2024-04-25T11:21:00.070606+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:21:00.070606+0000","last_undegraded":"2024-04-25T11:21:00.070606+0000","last_fullsized":"2024-04-25T11:21:00.070606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069701+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:21:00.069701+0000","last_peered":"2024-04-25T11:21:00.069701+0000","last_clean":"2024-04-25T11:21:00.069701+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:21:00.069701+0000","last_undegraded":"2024-04-25T11:21:00.069701+0000","last_fullsized":"2024-04-25T11:21:00.069701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110638+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:21:00.110638+0000","last_peered":"2024-04-25T11:21:00.110638+0000","last_clean":"2024-04-25T11:21:00.110638+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:21:00.110638+0000","last_undegraded":"2024-04-25T11:21:00.110638+0000","last_fullsized":"2024-04-25T11:21:00.110638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.957738+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:21:05.957738+0000","last_peered":"2024-04-25T11:21:05.957738+0000","last_clean":"2024-04-25T11:21:05.957738+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:21:05.957738+0000","last_undegraded":"2024-04-25T11:21:05.957738+0000","last_fullsized":"2024-04-25T11:21:05.957738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939656+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:59.939656+0000","last_peered":"2024-04-25T11:20:59.939656+0000","last_clean":"2024-04-25T11:20:59.939656+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:59.939656+0000","last_undegraded":"2024-04-25T11:20:59.939656+0000","last_fullsized":"2024-04-25T11:20:59.939656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924486+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:59.924486+0000","last_peered":"2024-04-25T11:20:59.924486+0000","last_clean":"2024-04-25T11:20:59.924486+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:59.924486+0000","last_undegraded":"2024-04-25T11:20:59.924486+0000","last_fullsized":"2024-04-25T11:20:59.924486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961276+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:21:05.961276+0000","last_peered":"2024-04-25T11:21:05.961276+0000","last_clean":"2024-04-25T11:21:05.961276+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:21:05.961276+0000","last_undegraded":"2024-04-25T11:21:05.961276+0000","last_fullsized":"2024-04-25T11:21:05.961276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931983+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:59.931983+0000","last_peered":"2024-04-25T11:20:59.931983+0000","last_clean":"2024-04-25T11:20:59.931983+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:59.931983+0000","last_undegraded":"2024-04-25T11:20:59.931983+0000","last_fullsized":"2024-04-25T11:20:59.931983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110991+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:21:00.110991+0000","last_peered":"2024-04-25T11:21:00.110991+0000","last_clean":"2024-04-25T11:21:00.110991+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:21:00.110991+0000","last_undegraded":"2024-04-25T11:21:00.110991+0000","last_fullsized":"2024-04-25T11:21:00.110991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959522+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:21:05.959522+0000","last_peered":"2024-04-25T11:21:05.959522+0000","last_clean":"2024-04-25T11:21:05.959522+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:21:05.959522+0000","last_undegraded":"2024-04-25T11:21:05.959522+0000","last_fullsized":"2024-04-25T11:21:05.959522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930980+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:59.930980+0000","last_peered":"2024-04-25T11:20:59.930980+0000","last_clean":"2024-04-25T11:20:59.930980+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:59.930980+0000","last_undegraded":"2024-04-25T11:20:59.930980+0000","last_fullsized":"2024-04-25T11:20:59.930980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935426+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:59.935426+0000","last_peered":"2024-04-25T11:20:59.935426+0000","last_clean":"2024-04-25T11:20:59.935426+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:59.935426+0000","last_undegraded":"2024-04-25T11:20:59.935426+0000","last_fullsized":"2024-04-25T11:20:59.935426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992627+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:21:11.992627+0000","last_peered":"2024-04-25T11:21:11.992627+0000","last_clean":"2024-04-25T11:21:11.992627+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:21:11.992627+0000","last_undegraded":"2024-04-25T11:21:11.992627+0000","last_fullsized":"2024-04-25T11:21:11.992627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476378+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:21:00.476378+0000","last_peered":"2024-04-25T11:21:00.476378+0000","last_clean":"2024-04-25T11:21:00.476378+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:21:00.476378+0000","last_undegraded":"2024-04-25T11:21:00.476378+0000","last_fullsized":"2024-04-25T11:21:00.476378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109709+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:21:00.109709+0000","last_peered":"2024-04-25T11:21:00.109709+0000","last_clean":"2024-04-25T11:21:00.109709+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:21:00.109709+0000","last_undegraded":"2024-04-25T11:21:00.109709+0000","last_fullsized":"2024-04-25T11:21:00.109709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'14","reported_seq":40,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002126+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:21:12.002126+0000","last_peered":"2024-04-25T11:21:12.002126+0000","last_clean":"2024-04-25T11:21:12.002126+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:21:12.002126+0000","last_undegraded":"2024-04-25T11:21:12.002126+0000","last_fullsized":"2024-04-25T11:21:12.002126+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936662+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:59.936662+0000","last_peered":"2024-04-25T11:20:59.936662+0000","last_clean":"2024-04-25T11:20:59.936662+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:59.936662+0000","last_undegraded":"2024-04-25T11:20:59.936662+0000","last_fullsized":"2024-04-25T11:20:59.936662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070136+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:21:00.070136+0000","last_peered":"2024-04-25T11:21:00.070136+0000","last_clean":"2024-04-25T11:21:00.070136+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:21:00.070136+0000","last_undegraded":"2024-04-25T11:21:00.070136+0000","last_fullsized":"2024-04-25T11:21:00.070136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938676+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:59.938676+0000","last_peered":"2024-04-25T11:20:59.938676+0000","last_clean":"2024-04-25T11:20:59.938676+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:59.938676+0000","last_undegraded":"2024-04-25T11:20:59.938676+0000","last_fullsized":"2024-04-25T11:20:59.938676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069494+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:21:00.069494+0000","last_peered":"2024-04-25T11:21:00.069494+0000","last_clean":"2024-04-25T11:21:00.069494+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:21:00.069494+0000","last_undegraded":"2024-04-25T11:21:00.069494+0000","last_fullsized":"2024-04-25T11:21:00.069494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940805+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:59.940805+0000","last_peered":"2024-04-25T11:20:59.940805+0000","last_clean":"2024-04-25T11:20:59.940805+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:59.940805+0000","last_undegraded":"2024-04-25T11:20:59.940805+0000","last_fullsized":"2024-04-25T11:20:59.940805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.997669+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:21:11.997669+0000","last_peered":"2024-04-25T11:21:11.997669+0000","last_clean":"2024-04-25T11:21:11.997669+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:21:11.997669+0000","last_undegraded":"2024-04-25T11:21:11.997669+0000","last_fullsized":"2024-04-25T11:21:11.997669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.073978+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:21:00.073978+0000","last_peered":"2024-04-25T11:21:00.073978+0000","last_clean":"2024-04-25T11:21:00.073978+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:21:00.073978+0000","last_undegraded":"2024-04-25T11:21:00.073978+0000","last_fullsized":"2024-04-25T11:21:00.073978+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476734+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:21:00.476734+0000","last_peered":"2024-04-25T11:21:00.476734+0000","last_clean":"2024-04-25T11:21:00.476734+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:21:00.476734+0000","last_undegraded":"2024-04-25T11:21:00.476734+0000","last_fullsized":"2024-04-25T11:21:00.476734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960966+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:21:05.960966+0000","last_peered":"2024-04-25T11:21:05.960966+0000","last_clean":"2024-04-25T11:21:05.960966+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:21:05.960966+0000","last_undegraded":"2024-04-25T11:21:05.960966+0000","last_fullsized":"2024-04-25T11:21:05.960966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928485+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:59.928485+0000","last_peered":"2024-04-25T11:20:59.928485+0000","last_clean":"2024-04-25T11:20:59.928485+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:59.928485+0000","last_undegraded":"2024-04-25T11:20:59.928485+0000","last_fullsized":"2024-04-25T11:20:59.928485+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938424+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:59.938424+0000","last_peered":"2024-04-25T11:20:59.938424+0000","last_clean":"2024-04-25T11:20:59.938424+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:59.938424+0000","last_undegraded":"2024-04-25T11:20:59.938424+0000","last_fullsized":"2024-04-25T11:20:59.938424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.944101+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:20:59.944101+0000","last_peered":"2024-04-25T11:20:59.944101+0000","last_clean":"2024-04-25T11:20:59.944101+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:20:59.944101+0000","last_undegraded":"2024-04-25T11:20:59.944101+0000","last_fullsized":"2024-04-25T11:20:59.944101+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.958897+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:21:05.958897+0000","last_peered":"2024-04-25T11:21:05.958897+0000","last_clean":"2024-04-25T11:21:05.958897+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:21:05.958897+0000","last_undegraded":"2024-04-25T11:21:05.958897+0000","last_fullsized":"2024-04-25T11:21:05.958897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924911+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:59.924911+0000","last_peered":"2024-04-25T11:20:59.924911+0000","last_clean":"2024-04-25T11:20:59.924911+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:59.924911+0000","last_undegraded":"2024-04-25T11:20:59.924911+0000","last_fullsized":"2024-04-25T11:20:59.924911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926423+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:59.926423+0000","last_peered":"2024-04-25T11:20:59.926423+0000","last_clean":"2024-04-25T11:20:59.926423+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:59.926423+0000","last_undegraded":"2024-04-25T11:20:59.926423+0000","last_fullsized":"2024-04-25T11:20:59.926423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.911858+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:21:00.911858+0000","last_peered":"2024-04-25T11:21:00.911858+0000","last_clean":"2024-04-25T11:21:00.911858+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:21:00.911858+0000","last_undegraded":"2024-04-25T11:21:00.911858+0000","last_fullsized":"2024-04-25T11:21:00.911858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961666+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:21:05.961666+0000","last_peered":"2024-04-25T11:21:05.961666+0000","last_clean":"2024-04-25T11:21:05.961666+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:21:05.961666+0000","last_undegraded":"2024-04-25T11:21:05.961666+0000","last_fullsized":"2024-04-25T11:21:05.961666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936830+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:59.936830+0000","last_peered":"2024-04-25T11:20:59.936830+0000","last_clean":"2024-04-25T11:20:59.936830+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:59.936830+0000","last_undegraded":"2024-04-25T11:20:59.936830+0000","last_fullsized":"2024-04-25T11:20:59.936830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924523+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:59.924523+0000","last_peered":"2024-04-25T11:20:59.924523+0000","last_clean":"2024-04-25T11:20:59.924523+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:59.924523+0000","last_undegraded":"2024-04-25T11:20:59.924523+0000","last_fullsized":"2024-04-25T11:20:59.924523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931078+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:59.931078+0000","last_peered":"2024-04-25T11:20:59.931078+0000","last_clean":"2024-04-25T11:20:59.931078+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:59.931078+0000","last_undegraded":"2024-04-25T11:20:59.931078+0000","last_fullsized":"2024-04-25T11:20:59.931078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067891+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:21:00.067891+0000","last_peered":"2024-04-25T11:21:00.067891+0000","last_clean":"2024-04-25T11:21:00.067891+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:21:00.067891+0000","last_undegraded":"2024-04-25T11:21:00.067891+0000","last_fullsized":"2024-04-25T11:21:00.067891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931524+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:59.931524+0000","last_peered":"2024-04-25T11:20:59.931524+0000","last_clean":"2024-04-25T11:20:59.931524+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:59.931524+0000","last_undegraded":"2024-04-25T11:20:59.931524+0000","last_fullsized":"2024-04-25T11:20:59.931524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924055+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:59.924055+0000","last_peered":"2024-04-25T11:20:59.924055+0000","last_clean":"2024-04-25T11:20:59.924055+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:59.924055+0000","last_undegraded":"2024-04-25T11:20:59.924055+0000","last_fullsized":"2024-04-25T11:20:59.924055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924236+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:59.924236+0000","last_peered":"2024-04-25T11:20:59.924236+0000","last_clean":"2024-04-25T11:20:59.924236+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:59.924236+0000","last_undegraded":"2024-04-25T11:20:59.924236+0000","last_fullsized":"2024-04-25T11:20:59.924236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474378+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:21:00.474378+0000","last_peered":"2024-04-25T11:21:00.474378+0000","last_clean":"2024-04-25T11:21:00.474378+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:21:00.474378+0000","last_undegraded":"2024-04-25T11:21:00.474378+0000","last_fullsized":"2024-04-25T11:21:00.474378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936557+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:59.936557+0000","last_peered":"2024-04-25T11:20:59.936557+0000","last_clean":"2024-04-25T11:20:59.936557+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:59.936557+0000","last_undegraded":"2024-04-25T11:20:59.936557+0000","last_fullsized":"2024-04-25T11:20:59.936557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931409+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:59.931409+0000","last_peered":"2024-04-25T11:20:59.931409+0000","last_clean":"2024-04-25T11:20:59.931409+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:59.931409+0000","last_undegraded":"2024-04-25T11:20:59.931409+0000","last_fullsized":"2024-04-25T11:20:59.931409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938092+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:59.938092+0000","last_peered":"2024-04-25T11:20:59.938092+0000","last_clean":"2024-04-25T11:20:59.938092+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:59.938092+0000","last_undegraded":"2024-04-25T11:20:59.938092+0000","last_fullsized":"2024-04-25T11:20:59.938092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924998+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:59.924998+0000","last_peered":"2024-04-25T11:20:59.924998+0000","last_clean":"2024-04-25T11:20:59.924998+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:59.924998+0000","last_undegraded":"2024-04-25T11:20:59.924998+0000","last_fullsized":"2024-04-25T11:20:59.924998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932179+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:59.932179+0000","last_peered":"2024-04-25T11:20:59.932179+0000","last_clean":"2024-04-25T11:20:59.932179+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:59.932179+0000","last_undegraded":"2024-04-25T11:20:59.932179+0000","last_fullsized":"2024-04-25T11:20:59.932179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474317+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:21:00.474317+0000","last_peered":"2024-04-25T11:21:00.474317+0000","last_clean":"2024-04-25T11:21:00.474317+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:21:00.474317+0000","last_undegraded":"2024-04-25T11:21:00.474317+0000","last_fullsized":"2024-04-25T11:21:00.474317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929163+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:59.929163+0000","last_peered":"2024-04-25T11:20:59.929163+0000","last_clean":"2024-04-25T11:20:59.929163+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:59.929163+0000","last_undegraded":"2024-04-25T11:20:59.929163+0000","last_fullsized":"2024-04-25T11:20:59.929163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.475936+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:21:00.475936+0000","last_peered":"2024-04-25T11:21:00.475936+0000","last_clean":"2024-04-25T11:21:00.475936+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:21:00.475936+0000","last_undegraded":"2024-04-25T11:21:00.475936+0000","last_fullsized":"2024-04-25T11:21:00.475936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936290+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:59.936290+0000","last_peered":"2024-04-25T11:20:59.936290+0000","last_clean":"2024-04-25T11:20:59.936290+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:59.936290+0000","last_undegraded":"2024-04-25T11:20:59.936290+0000","last_fullsized":"2024-04-25T11:20:59.936290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924776+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:59.924776+0000","last_peered":"2024-04-25T11:20:59.924776+0000","last_clean":"2024-04-25T11:20:59.924776+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:59.924776+0000","last_undegraded":"2024-04-25T11:20:59.924776+0000","last_fullsized":"2024-04-25T11:20:59.924776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.945079+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:20:59.945079+0000","last_peered":"2024-04-25T11:20:59.945079+0000","last_clean":"2024-04-25T11:20:59.945079+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:20:59.945079+0000","last_undegraded":"2024-04-25T11:20:59.945079+0000","last_fullsized":"2024-04-25T11:20:59.945079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926127+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:59.926127+0000","last_peered":"2024-04-25T11:20:59.926127+0000","last_clean":"2024-04-25T11:20:59.926127+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:59.926127+0000","last_undegraded":"2024-04-25T11:20:59.926127+0000","last_fullsized":"2024-04-25T11:20:59.926127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.477014+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:21:00.477014+0000","last_peered":"2024-04-25T11:21:00.477014+0000","last_clean":"2024-04-25T11:21:00.477014+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:21:00.477014+0000","last_undegraded":"2024-04-25T11:21:00.477014+0000","last_fullsized":"2024-04-25T11:21:00.477014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474387+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474387+0000","last_peered":"2024-04-25T11:21:00.474387+0000","last_clean":"2024-04-25T11:21:00.474387+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:21:00.474387+0000","last_undegraded":"2024-04-25T11:21:00.474387+0000","last_fullsized":"2024-04-25T11:21:00.474387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933616+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:59.933616+0000","last_peered":"2024-04-25T11:20:59.933616+0000","last_clean":"2024-04-25T11:20:59.933616+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:59.933616+0000","last_undegraded":"2024-04-25T11:20:59.933616+0000","last_fullsized":"2024-04-25T11:20:59.933616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935312+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:59.935312+0000","last_peered":"2024-04-25T11:20:59.935312+0000","last_clean":"2024-04-25T11:20:59.935312+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:59.935312+0000","last_undegraded":"2024-04-25T11:20:59.935312+0000","last_fullsized":"2024-04-25T11:20:59.935312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932719+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:59.932719+0000","last_peered":"2024-04-25T11:20:59.932719+0000","last_clean":"2024-04-25T11:20:59.932719+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:59.932719+0000","last_undegraded":"2024-04-25T11:20:59.932719+0000","last_fullsized":"2024-04-25T11:20:59.932719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068007+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:21:00.068007+0000","last_peered":"2024-04-25T11:21:00.068007+0000","last_clean":"2024-04-25T11:21:00.068007+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:21:00.068007+0000","last_undegraded":"2024-04-25T11:21:00.068007+0000","last_fullsized":"2024-04-25T11:21:00.068007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925200+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:59.925200+0000","last_peered":"2024-04-25T11:20:59.925200+0000","last_clean":"2024-04-25T11:20:59.925200+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:59.925200+0000","last_undegraded":"2024-04-25T11:20:59.925200+0000","last_fullsized":"2024-04-25T11:20:59.925200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924837+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:59.924837+0000","last_peered":"2024-04-25T11:20:59.924837+0000","last_clean":"2024-04-25T11:20:59.924837+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:59.924837+0000","last_undegraded":"2024-04-25T11:20:59.924837+0000","last_fullsized":"2024-04-25T11:20:59.924837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111746+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:21:00.111746+0000","last_peered":"2024-04-25T11:21:00.111746+0000","last_clean":"2024-04-25T11:21:00.111746+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:21:00.111746+0000","last_undegraded":"2024-04-25T11:21:00.111746+0000","last_fullsized":"2024-04-25T11:21:00.111746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926148+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:59.926148+0000","last_peered":"2024-04-25T11:20:59.926148+0000","last_clean":"2024-04-25T11:20:59.926148+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:59.926148+0000","last_undegraded":"2024-04-25T11:20:59.926148+0000","last_fullsized":"2024-04-25T11:20:59.926148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109471+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:21:00.109471+0000","last_peered":"2024-04-25T11:21:00.109471+0000","last_clean":"2024-04-25T11:21:00.109471+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:21:00.109471+0000","last_undegraded":"2024-04-25T11:21:00.109471+0000","last_fullsized":"2024-04-25T11:21:00.109471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474518+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:21:00.474518+0000","last_peered":"2024-04-25T11:21:00.474518+0000","last_clean":"2024-04-25T11:21:00.474518+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:21:00.474518+0000","last_undegraded":"2024-04-25T11:21:00.474518+0000","last_fullsized":"2024-04-25T11:21:00.474518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109512+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:21:00.109512+0000","last_peered":"2024-04-25T11:21:00.109512+0000","last_clean":"2024-04-25T11:21:00.109512+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:21:00.109512+0000","last_undegraded":"2024-04-25T11:21:00.109512+0000","last_fullsized":"2024-04-25T11:21:00.109512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474969+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:21:00.474969+0000","last_peered":"2024-04-25T11:21:00.474969+0000","last_clean":"2024-04-25T11:21:00.474969+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:21:00.474969+0000","last_undegraded":"2024-04-25T11:21:00.474969+0000","last_fullsized":"2024-04-25T11:21:00.474969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938661+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:59.938661+0000","last_peered":"2024-04-25T11:20:59.938661+0000","last_clean":"2024-04-25T11:20:59.938661+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:59.938661+0000","last_undegraded":"2024-04-25T11:20:59.938661+0000","last_fullsized":"2024-04-25T11:20:59.938661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939739+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:59.939739+0000","last_peered":"2024-04-25T11:20:59.939739+0000","last_clean":"2024-04-25T11:20:59.939739+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:59.939739+0000","last_undegraded":"2024-04-25T11:20:59.939739+0000","last_fullsized":"2024-04-25T11:20:59.939739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109891+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:21:00.109891+0000","last_peered":"2024-04-25T11:21:00.109891+0000","last_clean":"2024-04-25T11:21:00.109891+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:21:00.109891+0000","last_undegraded":"2024-04-25T11:21:00.109891+0000","last_fullsized":"2024-04-25T11:21:00.109891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":11587,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":214,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":23615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":706829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":118993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":118293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":115691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":119027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":115823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":115893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":901120,"data_stored":705603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:21:26.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:21:26.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 Processor -- start 2024-04-25T11:21:26.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 -- start start 2024-04-25T11:21:26.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ea40c8cd0 0x7f3ea414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ea40ce260 0x7f3ea40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea40cbad0 con 0x7f3ea40ca010 2024-04-25T11:21:26.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea40c8550 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea9315640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea40c7c20 con 0x7f3ea40ce260 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ea40c8cd0 0x7f3ea414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ea40c8cd0 0x7f3ea414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37090/0 (socket says 172.21.15.136:37090) 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/919731779 learned_addr learned my addr 172.21.15.136:0/919731779 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea27fc640 1 --2- 172.21.15.136:0/919731779 >> v2:172.21.15.157:3300/0 conn(0x7f3ea40ce260 0x7f3ea40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/919731779 >> v2:172.21.15.157:3300/0 conn(0x7f3ea40ce260 msgr2=0x7f3ea40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 --2- 172.21.15.136:0/919731779 >> v2:172.21.15.157:3300/0 conn(0x7f3ea40ce260 0x7f3ea40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/919731779 >> v1:172.21.15.157:6789/0 conn(0x7f3ea40ca010 legacy=0x7f3ea40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:26.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.811+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/919731779 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea40c7f00 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea2ffd640 1 --2- 172.21.15.136:0/919731779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea414f680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40c86d0 tx=0x7f3e8c0213a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860f29e882d8790 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea1ffb640 1 -- 172.21.15.136:0/919731779 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0b6070 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea1ffb640 1 -- 172.21.15.136:0/919731779 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0b1600 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea1ffb640 1 -- 172.21.15.136:0/919731779 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0b19d0 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 -- 172.21.15.136:0/919731779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 msgr2=0x7f3ea414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 --2- 172.21.15.136:0/919731779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea414f680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40c86d0 tx=0x7f3e8c0213a0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 -- 172.21.15.136:0/919731779 shutdown_connections 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 --2- 172.21.15.136:0/919731779 >> v2:172.21.15.157:3300/0 conn(0x7f3ea40ce260 0x7f3ea40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 --2- 172.21.15.136:0/919731779 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea414f680 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 -- 172.21.15.136:0/919731779 >> 172.21.15.136:0/919731779 conn(0x7f3ea406f9d0 msgr2=0x7f3ea406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 -- 172.21.15.136:0/919731779 shutdown_connections 2024-04-25T11:21:26.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.815+0000 7f3ea9315640 1 -- 172.21.15.136:0/919731779 wait complete. 2024-04-25T11:21:26.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 Processor -- start 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- start start 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea40ca010 0x7f3ea4153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ea40ce260 0x7f3ea413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea40cb720 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea414fbc0 con 0x7f3ea40ca010 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea406ded0 con 0x7f3ea40ce260 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37092/0 (socket says 172.21.15.136:37092) 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/1286919163 learned_addr learned my addr 172.21.15.136:0/1286919163 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea27fc640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea40ca010 0x7f3ea4153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea37fe640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ea40ce260 0x7f3ea413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ea40ce260 msgr2=0x7f3ea413cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ea40ce260 0x7f3ea413cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea40ca010 msgr2=0x7f3ea4153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea40ca010 0x7f3ea4153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 -- 172.21.15.136:0/1286919163 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea41f6af0 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea2ffd640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00b2c0 tx=0x7f3e8c01fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0c1070 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0b07b0 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0b6070 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea4154d50 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.819+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea4155310 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea406a350 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3e8c0c4070 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3e9b7fe640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 0x7f3e84071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:26.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3e8c12cf20 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3ea27fc640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 0x7f3e84071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:26.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.823+0000 7f3ea27fc640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 0x7f3e84071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3ea414e2b0 tx=0x7f3e94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:26.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.827+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3e8c01af00 con 0x7f3ea40c8cd0 2024-04-25T11:21:26.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.971+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ea4154b00 con 0x7f3e8406f400 2024-04-25T11:21:26.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.995+0000 7f3e9b7fe640 1 -- 172.21.15.136:0/1286919163 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332648 (secure 0 0 0) 0x7f3ea4154b00 con 0x7f3e8406f400 2024-04-25T11:21:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:26.999 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 msgr2=0x7f3e84071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 0x7f3e84071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3ea414e2b0 tx=0x7f3e94008040 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 msgr2=0x7f3ea41f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00b2c0 tx=0x7f3e8c01fbe0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 shutdown_connections 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3e8406f400 0x7f3e84071910 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ea40ce260 0x7f3ea413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea40ca010 0x7f3ea4153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 --2- 172.21.15.136:0/1286919163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ea40c8cd0 0x7f3ea41f65b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 >> 172.21.15.136:0/1286919163 conn(0x7f3ea406f9d0 msgr2=0x7f3ea40cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 shutdown_connections 2024-04-25T11:21:27.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:26.999+0000 7f3ea9315640 1 -- 172.21.15.136:0/1286919163 wait complete. 2024-04-25T11:21:27.016 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-25T11:21:26.779999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602396,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":317,"num_write_kb":1702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236360,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738360,"statfs":{"total":773094113280,"available":772852080640,"internally_reserved":0,"allocated":3678208,"data_stored":2119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3170,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005169"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.994195+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:21:11.994195+0000","last_peered":"2024-04-25T11:21:11.994195+0000","last_clean":"2024-04-25T11:21:11.994195+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:21:11.994195+0000","last_undegraded":"2024-04-25T11:21:11.994195+0000","last_fullsized":"2024-04-25T11:21:11.994195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931730+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:59.931730+0000","last_peered":"2024-04-25T11:20:59.931730+0000","last_clean":"2024-04-25T11:20:59.931730+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:59.931730+0000","last_undegraded":"2024-04-25T11:20:59.931730+0000","last_fullsized":"2024-04-25T11:20:59.931730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928414+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:59.928414+0000","last_peered":"2024-04-25T11:20:59.928414+0000","last_clean":"2024-04-25T11:20:59.928414+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:59.928414+0000","last_undegraded":"2024-04-25T11:20:59.928414+0000","last_fullsized":"2024-04-25T11:20:59.928414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932201+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:59.932201+0000","last_peered":"2024-04-25T11:20:59.932201+0000","last_clean":"2024-04-25T11:20:59.932201+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:59.932201+0000","last_undegraded":"2024-04-25T11:20:59.932201+0000","last_fullsized":"2024-04-25T11:20:59.932201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940417+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:59.940417+0000","last_peered":"2024-04-25T11:20:59.940417+0000","last_clean":"2024-04-25T11:20:59.940417+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:59.940417+0000","last_undegraded":"2024-04-25T11:20:59.940417+0000","last_fullsized":"2024-04-25T11:20:59.940417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068980+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:21:00.068980+0000","last_peered":"2024-04-25T11:21:00.068980+0000","last_clean":"2024-04-25T11:21:00.068980+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:21:00.068980+0000","last_undegraded":"2024-04-25T11:21:00.068980+0000","last_fullsized":"2024-04-25T11:21:00.068980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.003480+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:21:12.003480+0000","last_peered":"2024-04-25T11:21:12.003480+0000","last_clean":"2024-04-25T11:21:12.003480+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:12.003480+0000","last_undegraded":"2024-04-25T11:21:12.003480+0000","last_fullsized":"2024-04-25T11:21:12.003480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075805+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:21:00.075805+0000","last_peered":"2024-04-25T11:21:00.075805+0000","last_clean":"2024-04-25T11:21:00.075805+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:21:00.075805+0000","last_undegraded":"2024-04-25T11:21:00.075805+0000","last_fullsized":"2024-04-25T11:21:00.075805+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069307+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:21:00.069307+0000","last_peered":"2024-04-25T11:21:00.069307+0000","last_clean":"2024-04-25T11:21:00.069307+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:21:00.069307+0000","last_undegraded":"2024-04-25T11:21:00.069307+0000","last_fullsized":"2024-04-25T11:21:00.069307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002064+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:21:12.002064+0000","last_peered":"2024-04-25T11:21:12.002064+0000","last_clean":"2024-04-25T11:21:12.002064+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:21:12.002064+0000","last_undegraded":"2024-04-25T11:21:12.002064+0000","last_fullsized":"2024-04-25T11:21:12.002064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075553+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:21:00.075553+0000","last_peered":"2024-04-25T11:21:00.075553+0000","last_clean":"2024-04-25T11:21:00.075553+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:21:00.075553+0000","last_undegraded":"2024-04-25T11:21:00.075553+0000","last_fullsized":"2024-04-25T11:21:00.075553+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069045+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:21:00.069045+0000","last_peered":"2024-04-25T11:21:00.069045+0000","last_clean":"2024-04-25T11:21:00.069045+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:21:00.069045+0000","last_undegraded":"2024-04-25T11:21:00.069045+0000","last_fullsized":"2024-04-25T11:21:00.069045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924874+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:59.924874+0000","last_peered":"2024-04-25T11:20:59.924874+0000","last_clean":"2024-04-25T11:20:59.924874+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:59.924874+0000","last_undegraded":"2024-04-25T11:20:59.924874+0000","last_fullsized":"2024-04-25T11:20:59.924874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111184+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:21:00.111184+0000","last_peered":"2024-04-25T11:21:00.111184+0000","last_clean":"2024-04-25T11:21:00.111184+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:21:00.111184+0000","last_undegraded":"2024-04-25T11:21:00.111184+0000","last_fullsized":"2024-04-25T11:21:00.111184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070128+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:21:00.070128+0000","last_peered":"2024-04-25T11:21:00.070128+0000","last_clean":"2024-04-25T11:21:00.070128+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:21:00.070128+0000","last_undegraded":"2024-04-25T11:21:00.070128+0000","last_fullsized":"2024-04-25T11:21:00.070128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930588+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:59.930588+0000","last_peered":"2024-04-25T11:20:59.930588+0000","last_clean":"2024-04-25T11:20:59.930588+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:59.930588+0000","last_undegraded":"2024-04-25T11:20:59.930588+0000","last_fullsized":"2024-04-25T11:20:59.930588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959714+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:21:05.959714+0000","last_peered":"2024-04-25T11:21:05.959714+0000","last_clean":"2024-04-25T11:21:05.959714+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:21:05.959714+0000","last_undegraded":"2024-04-25T11:21:05.959714+0000","last_fullsized":"2024-04-25T11:21:05.959714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069432+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:21:00.069432+0000","last_peered":"2024-04-25T11:21:00.069432+0000","last_clean":"2024-04-25T11:21:00.069432+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:21:00.069432+0000","last_undegraded":"2024-04-25T11:21:00.069432+0000","last_fullsized":"2024-04-25T11:21:00.069432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926697+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:59.926697+0000","last_peered":"2024-04-25T11:20:59.926697+0000","last_clean":"2024-04-25T11:20:59.926697+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:59.926697+0000","last_undegraded":"2024-04-25T11:20:59.926697+0000","last_fullsized":"2024-04-25T11:20:59.926697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.991983+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:21:11.991983+0000","last_peered":"2024-04-25T11:21:11.991983+0000","last_clean":"2024-04-25T11:21:11.991983+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:21:11.991983+0000","last_undegraded":"2024-04-25T11:21:11.991983+0000","last_fullsized":"2024-04-25T11:21:11.991983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.114813+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:21:00.114813+0000","last_peered":"2024-04-25T11:21:00.114813+0000","last_clean":"2024-04-25T11:21:00.114813+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:21:00.114813+0000","last_undegraded":"2024-04-25T11:21:00.114813+0000","last_fullsized":"2024-04-25T11:21:00.114813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067738+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:21:00.067738+0000","last_peered":"2024-04-25T11:21:00.067738+0000","last_clean":"2024-04-25T11:21:00.067738+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:21:00.067738+0000","last_undegraded":"2024-04-25T11:21:00.067738+0000","last_fullsized":"2024-04-25T11:21:00.067738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110139+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:21:00.110139+0000","last_peered":"2024-04-25T11:21:00.110139+0000","last_clean":"2024-04-25T11:21:00.110139+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:21:00.110139+0000","last_undegraded":"2024-04-25T11:21:00.110139+0000","last_fullsized":"2024-04-25T11:21:00.110139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070453+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:21:00.070453+0000","last_peered":"2024-04-25T11:21:00.070453+0000","last_clean":"2024-04-25T11:21:00.070453+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:21:00.070453+0000","last_undegraded":"2024-04-25T11:21:00.070453+0000","last_fullsized":"2024-04-25T11:21:00.070453+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940092+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:59.940092+0000","last_peered":"2024-04-25T11:20:59.940092+0000","last_clean":"2024-04-25T11:20:59.940092+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:59.940092+0000","last_undegraded":"2024-04-25T11:20:59.940092+0000","last_fullsized":"2024-04-25T11:20:59.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002572+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:21:12.002572+0000","last_peered":"2024-04-25T11:21:12.002572+0000","last_clean":"2024-04-25T11:21:12.002572+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:21:12.002572+0000","last_undegraded":"2024-04-25T11:21:12.002572+0000","last_fullsized":"2024-04-25T11:21:12.002572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069767+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:21:00.069767+0000","last_peered":"2024-04-25T11:21:00.069767+0000","last_clean":"2024-04-25T11:21:00.069767+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:21:00.069767+0000","last_undegraded":"2024-04-25T11:21:00.069767+0000","last_fullsized":"2024-04-25T11:21:00.069767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932437+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:59.932437+0000","last_peered":"2024-04-25T11:20:59.932437+0000","last_clean":"2024-04-25T11:20:59.932437+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:59.932437+0000","last_undegraded":"2024-04-25T11:20:59.932437+0000","last_fullsized":"2024-04-25T11:20:59.932437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962037+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:21:05.962037+0000","last_peered":"2024-04-25T11:21:05.962037+0000","last_clean":"2024-04-25T11:21:05.962037+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:21:05.962037+0000","last_undegraded":"2024-04-25T11:21:05.962037+0000","last_fullsized":"2024-04-25T11:21:05.962037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925818+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:59.925818+0000","last_peered":"2024-04-25T11:20:59.925818+0000","last_clean":"2024-04-25T11:20:59.925818+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:59.925818+0000","last_undegraded":"2024-04-25T11:20:59.925818+0000","last_fullsized":"2024-04-25T11:20:59.925818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936496+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:59.936496+0000","last_peered":"2024-04-25T11:20:59.936496+0000","last_clean":"2024-04-25T11:20:59.936496+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:59.936496+0000","last_undegraded":"2024-04-25T11:20:59.936496+0000","last_fullsized":"2024-04-25T11:20:59.936496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962135+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:21:05.962135+0000","last_peered":"2024-04-25T11:21:05.962135+0000","last_clean":"2024-04-25T11:21:05.962135+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:05.962135+0000","last_undegraded":"2024-04-25T11:21:05.962135+0000","last_fullsized":"2024-04-25T11:21:05.962135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935995+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:59.935995+0000","last_peered":"2024-04-25T11:20:59.935995+0000","last_clean":"2024-04-25T11:20:59.935995+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:59.935995+0000","last_undegraded":"2024-04-25T11:20:59.935995+0000","last_fullsized":"2024-04-25T11:20:59.935995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474622+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474622+0000","last_peered":"2024-04-25T11:21:00.474622+0000","last_clean":"2024-04-25T11:21:00.474622+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:21:00.474622+0000","last_undegraded":"2024-04-25T11:21:00.474622+0000","last_fullsized":"2024-04-25T11:21:00.474622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936918+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:59.936918+0000","last_peered":"2024-04-25T11:20:59.936918+0000","last_clean":"2024-04-25T11:20:59.936918+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:59.936918+0000","last_undegraded":"2024-04-25T11:20:59.936918+0000","last_fullsized":"2024-04-25T11:20:59.936918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936370+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:59.936370+0000","last_peered":"2024-04-25T11:20:59.936370+0000","last_clean":"2024-04-25T11:20:59.936370+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:59.936370+0000","last_undegraded":"2024-04-25T11:20:59.936370+0000","last_fullsized":"2024-04-25T11:20:59.936370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068094+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:21:00.068094+0000","last_peered":"2024-04-25T11:21:00.068094+0000","last_clean":"2024-04-25T11:21:00.068094+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:21:00.068094+0000","last_undegraded":"2024-04-25T11:21:00.068094+0000","last_fullsized":"2024-04-25T11:21:00.068094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.999818+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:21:11.999818+0000","last_peered":"2024-04-25T11:21:11.999818+0000","last_clean":"2024-04-25T11:21:11.999818+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:21:11.999818+0000","last_undegraded":"2024-04-25T11:21:11.999818+0000","last_fullsized":"2024-04-25T11:21:11.999818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:59.936416+0000","last_peered":"2024-04-25T11:20:59.936416+0000","last_clean":"2024-04-25T11:20:59.936416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:59.936416+0000","last_undegraded":"2024-04-25T11:20:59.936416+0000","last_fullsized":"2024-04-25T11:20:59.936416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.934845+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:59.934845+0000","last_peered":"2024-04-25T11:20:59.934845+0000","last_clean":"2024-04-25T11:20:59.934845+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:59.934845+0000","last_undegraded":"2024-04-25T11:20:59.934845+0000","last_fullsized":"2024-04-25T11:20:59.934845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992803+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:21:11.992803+0000","last_peered":"2024-04-25T11:21:11.992803+0000","last_clean":"2024-04-25T11:21:11.992803+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:21:11.992803+0000","last_undegraded":"2024-04-25T11:21:11.992803+0000","last_fullsized":"2024-04-25T11:21:11.992803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939030+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:59.939030+0000","last_peered":"2024-04-25T11:20:59.939030+0000","last_clean":"2024-04-25T11:20:59.939030+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:59.939030+0000","last_undegraded":"2024-04-25T11:20:59.939030+0000","last_fullsized":"2024-04-25T11:20:59.939030+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933545+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:59.933545+0000","last_peered":"2024-04-25T11:20:59.933545+0000","last_clean":"2024-04-25T11:20:59.933545+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:59.933545+0000","last_undegraded":"2024-04-25T11:20:59.933545+0000","last_fullsized":"2024-04-25T11:20:59.933545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.917110+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:21:00.917110+0000","last_peered":"2024-04-25T11:21:00.917110+0000","last_clean":"2024-04-25T11:21:00.917110+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:21:00.917110+0000","last_undegraded":"2024-04-25T11:21:00.917110+0000","last_fullsized":"2024-04-25T11:21:00.917110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939772+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:59.939772+0000","last_peered":"2024-04-25T11:20:59.939772+0000","last_clean":"2024-04-25T11:20:59.939772+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:59.939772+0000","last_undegraded":"2024-04-25T11:20:59.939772+0000","last_fullsized":"2024-04-25T11:20:59.939772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929859+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:59.929859+0000","last_peered":"2024-04-25T11:20:59.929859+0000","last_clean":"2024-04-25T11:20:59.929859+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:59.929859+0000","last_undegraded":"2024-04-25T11:20:59.929859+0000","last_fullsized":"2024-04-25T11:20:59.929859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960397+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:21:05.960397+0000","last_peered":"2024-04-25T11:21:05.960397+0000","last_clean":"2024-04-25T11:21:05.960397+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:21:05.960397+0000","last_undegraded":"2024-04-25T11:21:05.960397+0000","last_fullsized":"2024-04-25T11:21:05.960397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":122,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110312+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:21:00.110312+0000","last_peered":"2024-04-25T11:21:00.110312+0000","last_clean":"2024-04-25T11:21:00.110312+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:21:00.110312+0000","last_undegraded":"2024-04-25T11:21:00.110312+0000","last_fullsized":"2024-04-25T11:21:00.110312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924917+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:59.924917+0000","last_peered":"2024-04-25T11:20:59.924917+0000","last_clean":"2024-04-25T11:20:59.924917+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:59.924917+0000","last_undegraded":"2024-04-25T11:20:59.924917+0000","last_fullsized":"2024-04-25T11:20:59.924917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111023+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:21:00.111023+0000","last_peered":"2024-04-25T11:21:00.111023+0000","last_clean":"2024-04-25T11:21:00.111023+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:21:00.111023+0000","last_undegraded":"2024-04-25T11:21:00.111023+0000","last_fullsized":"2024-04-25T11:21:00.111023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961234+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:21:05.961234+0000","last_peered":"2024-04-25T11:21:05.961234+0000","last_clean":"2024-04-25T11:21:05.961234+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:21:05.961234+0000","last_undegraded":"2024-04-25T11:21:05.961234+0000","last_fullsized":"2024-04-25T11:21:05.961234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924269+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:59.924269+0000","last_peered":"2024-04-25T11:20:59.924269+0000","last_clean":"2024-04-25T11:20:59.924269+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:59.924269+0000","last_undegraded":"2024-04-25T11:20:59.924269+0000","last_fullsized":"2024-04-25T11:20:59.924269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474615+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:21:00.474615+0000","last_peered":"2024-04-25T11:21:00.474615+0000","last_clean":"2024-04-25T11:21:00.474615+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:21:00.474615+0000","last_undegraded":"2024-04-25T11:21:00.474615+0000","last_fullsized":"2024-04-25T11:21:00.474615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961055+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:21:05.961055+0000","last_peered":"2024-04-25T11:21:05.961055+0000","last_clean":"2024-04-25T11:21:05.961055+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:21:05.961055+0000","last_undegraded":"2024-04-25T11:21:05.961055+0000","last_fullsized":"2024-04-25T11:21:05.961055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.479066+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:21:00.479066+0000","last_peered":"2024-04-25T11:21:00.479066+0000","last_clean":"2024-04-25T11:21:00.479066+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:21:00.479066+0000","last_undegraded":"2024-04-25T11:21:00.479066+0000","last_fullsized":"2024-04-25T11:21:00.479066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111834+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:21:00.111834+0000","last_peered":"2024-04-25T11:21:00.111834+0000","last_clean":"2024-04-25T11:21:00.111834+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:21:00.111834+0000","last_undegraded":"2024-04-25T11:21:00.111834+0000","last_fullsized":"2024-04-25T11:21:00.111834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992924+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:21:11.992924+0000","last_peered":"2024-04-25T11:21:11.992924+0000","last_clean":"2024-04-25T11:21:11.992924+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:21:11.992924+0000","last_undegraded":"2024-04-25T11:21:11.992924+0000","last_fullsized":"2024-04-25T11:21:11.992924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474787+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:21:00.474787+0000","last_peered":"2024-04-25T11:21:00.474787+0000","last_clean":"2024-04-25T11:21:00.474787+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:21:00.474787+0000","last_undegraded":"2024-04-25T11:21:00.474787+0000","last_fullsized":"2024-04-25T11:21:00.474787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111717+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:21:00.111717+0000","last_peered":"2024-04-25T11:21:00.111717+0000","last_clean":"2024-04-25T11:21:00.111717+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:21:00.111717+0000","last_undegraded":"2024-04-25T11:21:00.111717+0000","last_fullsized":"2024-04-25T11:21:00.111717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070606+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:21:00.070606+0000","last_peered":"2024-04-25T11:21:00.070606+0000","last_clean":"2024-04-25T11:21:00.070606+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:21:00.070606+0000","last_undegraded":"2024-04-25T11:21:00.070606+0000","last_fullsized":"2024-04-25T11:21:00.070606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069701+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:21:00.069701+0000","last_peered":"2024-04-25T11:21:00.069701+0000","last_clean":"2024-04-25T11:21:00.069701+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:21:00.069701+0000","last_undegraded":"2024-04-25T11:21:00.069701+0000","last_fullsized":"2024-04-25T11:21:00.069701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110638+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:21:00.110638+0000","last_peered":"2024-04-25T11:21:00.110638+0000","last_clean":"2024-04-25T11:21:00.110638+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:21:00.110638+0000","last_undegraded":"2024-04-25T11:21:00.110638+0000","last_fullsized":"2024-04-25T11:21:00.110638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.957738+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:21:05.957738+0000","last_peered":"2024-04-25T11:21:05.957738+0000","last_clean":"2024-04-25T11:21:05.957738+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:21:05.957738+0000","last_undegraded":"2024-04-25T11:21:05.957738+0000","last_fullsized":"2024-04-25T11:21:05.957738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939656+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:59.939656+0000","last_peered":"2024-04-25T11:20:59.939656+0000","last_clean":"2024-04-25T11:20:59.939656+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:59.939656+0000","last_undegraded":"2024-04-25T11:20:59.939656+0000","last_fullsized":"2024-04-25T11:20:59.939656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924486+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:59.924486+0000","last_peered":"2024-04-25T11:20:59.924486+0000","last_clean":"2024-04-25T11:20:59.924486+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:59.924486+0000","last_undegraded":"2024-04-25T11:20:59.924486+0000","last_fullsized":"2024-04-25T11:20:59.924486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961276+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:21:05.961276+0000","last_peered":"2024-04-25T11:21:05.961276+0000","last_clean":"2024-04-25T11:21:05.961276+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:21:05.961276+0000","last_undegraded":"2024-04-25T11:21:05.961276+0000","last_fullsized":"2024-04-25T11:21:05.961276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931983+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:59.931983+0000","last_peered":"2024-04-25T11:20:59.931983+0000","last_clean":"2024-04-25T11:20:59.931983+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:59.931983+0000","last_undegraded":"2024-04-25T11:20:59.931983+0000","last_fullsized":"2024-04-25T11:20:59.931983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110991+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:21:00.110991+0000","last_peered":"2024-04-25T11:21:00.110991+0000","last_clean":"2024-04-25T11:21:00.110991+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:21:00.110991+0000","last_undegraded":"2024-04-25T11:21:00.110991+0000","last_fullsized":"2024-04-25T11:21:00.110991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959522+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:21:05.959522+0000","last_peered":"2024-04-25T11:21:05.959522+0000","last_clean":"2024-04-25T11:21:05.959522+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:21:05.959522+0000","last_undegraded":"2024-04-25T11:21:05.959522+0000","last_fullsized":"2024-04-25T11:21:05.959522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930980+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:59.930980+0000","last_peered":"2024-04-25T11:20:59.930980+0000","last_clean":"2024-04-25T11:20:59.930980+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:59.930980+0000","last_undegraded":"2024-04-25T11:20:59.930980+0000","last_fullsized":"2024-04-25T11:20:59.930980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935426+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:59.935426+0000","last_peered":"2024-04-25T11:20:59.935426+0000","last_clean":"2024-04-25T11:20:59.935426+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:59.935426+0000","last_undegraded":"2024-04-25T11:20:59.935426+0000","last_fullsized":"2024-04-25T11:20:59.935426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992627+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:21:11.992627+0000","last_peered":"2024-04-25T11:21:11.992627+0000","last_clean":"2024-04-25T11:21:11.992627+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:21:11.992627+0000","last_undegraded":"2024-04-25T11:21:11.992627+0000","last_fullsized":"2024-04-25T11:21:11.992627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476378+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:21:00.476378+0000","last_peered":"2024-04-25T11:21:00.476378+0000","last_clean":"2024-04-25T11:21:00.476378+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:21:00.476378+0000","last_undegraded":"2024-04-25T11:21:00.476378+0000","last_fullsized":"2024-04-25T11:21:00.476378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109709+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:21:00.109709+0000","last_peered":"2024-04-25T11:21:00.109709+0000","last_clean":"2024-04-25T11:21:00.109709+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:21:00.109709+0000","last_undegraded":"2024-04-25T11:21:00.109709+0000","last_fullsized":"2024-04-25T11:21:00.109709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'16","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:20.836111+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:21:20.836111+0000","last_peered":"2024-04-25T11:21:20.836111+0000","last_clean":"2024-04-25T11:21:20.836111+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:21:20.836111+0000","last_undegraded":"2024-04-25T11:21:20.836111+0000","last_fullsized":"2024-04-25T11:21:20.836111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4283,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936662+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:59.936662+0000","last_peered":"2024-04-25T11:20:59.936662+0000","last_clean":"2024-04-25T11:20:59.936662+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:59.936662+0000","last_undegraded":"2024-04-25T11:20:59.936662+0000","last_fullsized":"2024-04-25T11:20:59.936662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070136+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:21:00.070136+0000","last_peered":"2024-04-25T11:21:00.070136+0000","last_clean":"2024-04-25T11:21:00.070136+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:21:00.070136+0000","last_undegraded":"2024-04-25T11:21:00.070136+0000","last_fullsized":"2024-04-25T11:21:00.070136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938676+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:59.938676+0000","last_peered":"2024-04-25T11:20:59.938676+0000","last_clean":"2024-04-25T11:20:59.938676+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:59.938676+0000","last_undegraded":"2024-04-25T11:20:59.938676+0000","last_fullsized":"2024-04-25T11:20:59.938676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069494+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:21:00.069494+0000","last_peered":"2024-04-25T11:21:00.069494+0000","last_clean":"2024-04-25T11:21:00.069494+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:21:00.069494+0000","last_undegraded":"2024-04-25T11:21:00.069494+0000","last_fullsized":"2024-04-25T11:21:00.069494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940805+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:59.940805+0000","last_peered":"2024-04-25T11:20:59.940805+0000","last_clean":"2024-04-25T11:20:59.940805+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:59.940805+0000","last_undegraded":"2024-04-25T11:20:59.940805+0000","last_fullsized":"2024-04-25T11:20:59.940805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.997669+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:21:11.997669+0000","last_peered":"2024-04-25T11:21:11.997669+0000","last_clean":"2024-04-25T11:21:11.997669+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:21:11.997669+0000","last_undegraded":"2024-04-25T11:21:11.997669+0000","last_fullsized":"2024-04-25T11:21:11.997669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.073978+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:21:00.073978+0000","last_peered":"2024-04-25T11:21:00.073978+0000","last_clean":"2024-04-25T11:21:00.073978+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:21:00.073978+0000","last_undegraded":"2024-04-25T11:21:00.073978+0000","last_fullsized":"2024-04-25T11:21:00.073978+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476734+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:21:00.476734+0000","last_peered":"2024-04-25T11:21:00.476734+0000","last_clean":"2024-04-25T11:21:00.476734+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:21:00.476734+0000","last_undegraded":"2024-04-25T11:21:00.476734+0000","last_fullsized":"2024-04-25T11:21:00.476734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960966+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:21:05.960966+0000","last_peered":"2024-04-25T11:21:05.960966+0000","last_clean":"2024-04-25T11:21:05.960966+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:21:05.960966+0000","last_undegraded":"2024-04-25T11:21:05.960966+0000","last_fullsized":"2024-04-25T11:21:05.960966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928485+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:59.928485+0000","last_peered":"2024-04-25T11:20:59.928485+0000","last_clean":"2024-04-25T11:20:59.928485+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:59.928485+0000","last_undegraded":"2024-04-25T11:20:59.928485+0000","last_fullsized":"2024-04-25T11:20:59.928485+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938424+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:59.938424+0000","last_peered":"2024-04-25T11:20:59.938424+0000","last_clean":"2024-04-25T11:20:59.938424+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:59.938424+0000","last_undegraded":"2024-04-25T11:20:59.938424+0000","last_fullsized":"2024-04-25T11:20:59.938424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.944101+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:20:59.944101+0000","last_peered":"2024-04-25T11:20:59.944101+0000","last_clean":"2024-04-25T11:20:59.944101+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:20:59.944101+0000","last_undegraded":"2024-04-25T11:20:59.944101+0000","last_fullsized":"2024-04-25T11:20:59.944101+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.958897+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:21:05.958897+0000","last_peered":"2024-04-25T11:21:05.958897+0000","last_clean":"2024-04-25T11:21:05.958897+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:21:05.958897+0000","last_undegraded":"2024-04-25T11:21:05.958897+0000","last_fullsized":"2024-04-25T11:21:05.958897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924911+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:59.924911+0000","last_peered":"2024-04-25T11:20:59.924911+0000","last_clean":"2024-04-25T11:20:59.924911+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:59.924911+0000","last_undegraded":"2024-04-25T11:20:59.924911+0000","last_fullsized":"2024-04-25T11:20:59.924911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926423+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:59.926423+0000","last_peered":"2024-04-25T11:20:59.926423+0000","last_clean":"2024-04-25T11:20:59.926423+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:59.926423+0000","last_undegraded":"2024-04-25T11:20:59.926423+0000","last_fullsized":"2024-04-25T11:20:59.926423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.911858+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:21:00.911858+0000","last_peered":"2024-04-25T11:21:00.911858+0000","last_clean":"2024-04-25T11:21:00.911858+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:21:00.911858+0000","last_undegraded":"2024-04-25T11:21:00.911858+0000","last_fullsized":"2024-04-25T11:21:00.911858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961666+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:21:05.961666+0000","last_peered":"2024-04-25T11:21:05.961666+0000","last_clean":"2024-04-25T11:21:05.961666+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:21:05.961666+0000","last_undegraded":"2024-04-25T11:21:05.961666+0000","last_fullsized":"2024-04-25T11:21:05.961666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936830+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:59.936830+0000","last_peered":"2024-04-25T11:20:59.936830+0000","last_clean":"2024-04-25T11:20:59.936830+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:59.936830+0000","last_undegraded":"2024-04-25T11:20:59.936830+0000","last_fullsized":"2024-04-25T11:20:59.936830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924523+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:59.924523+0000","last_peered":"2024-04-25T11:20:59.924523+0000","last_clean":"2024-04-25T11:20:59.924523+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:59.924523+0000","last_undegraded":"2024-04-25T11:20:59.924523+0000","last_fullsized":"2024-04-25T11:20:59.924523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931078+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:59.931078+0000","last_peered":"2024-04-25T11:20:59.931078+0000","last_clean":"2024-04-25T11:20:59.931078+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:59.931078+0000","last_undegraded":"2024-04-25T11:20:59.931078+0000","last_fullsized":"2024-04-25T11:20:59.931078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067891+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:21:00.067891+0000","last_peered":"2024-04-25T11:21:00.067891+0000","last_clean":"2024-04-25T11:21:00.067891+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:21:00.067891+0000","last_undegraded":"2024-04-25T11:21:00.067891+0000","last_fullsized":"2024-04-25T11:21:00.067891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931524+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:59.931524+0000","last_peered":"2024-04-25T11:20:59.931524+0000","last_clean":"2024-04-25T11:20:59.931524+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:59.931524+0000","last_undegraded":"2024-04-25T11:20:59.931524+0000","last_fullsized":"2024-04-25T11:20:59.931524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924055+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:59.924055+0000","last_peered":"2024-04-25T11:20:59.924055+0000","last_clean":"2024-04-25T11:20:59.924055+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:59.924055+0000","last_undegraded":"2024-04-25T11:20:59.924055+0000","last_fullsized":"2024-04-25T11:20:59.924055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924236+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:59.924236+0000","last_peered":"2024-04-25T11:20:59.924236+0000","last_clean":"2024-04-25T11:20:59.924236+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:59.924236+0000","last_undegraded":"2024-04-25T11:20:59.924236+0000","last_fullsized":"2024-04-25T11:20:59.924236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474378+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:21:00.474378+0000","last_peered":"2024-04-25T11:21:00.474378+0000","last_clean":"2024-04-25T11:21:00.474378+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:21:00.474378+0000","last_undegraded":"2024-04-25T11:21:00.474378+0000","last_fullsized":"2024-04-25T11:21:00.474378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936557+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:59.936557+0000","last_peered":"2024-04-25T11:20:59.936557+0000","last_clean":"2024-04-25T11:20:59.936557+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:59.936557+0000","last_undegraded":"2024-04-25T11:20:59.936557+0000","last_fullsized":"2024-04-25T11:20:59.936557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931409+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:59.931409+0000","last_peered":"2024-04-25T11:20:59.931409+0000","last_clean":"2024-04-25T11:20:59.931409+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:59.931409+0000","last_undegraded":"2024-04-25T11:20:59.931409+0000","last_fullsized":"2024-04-25T11:20:59.931409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938092+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:59.938092+0000","last_peered":"2024-04-25T11:20:59.938092+0000","last_clean":"2024-04-25T11:20:59.938092+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:59.938092+0000","last_undegraded":"2024-04-25T11:20:59.938092+0000","last_fullsized":"2024-04-25T11:20:59.938092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924998+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:59.924998+0000","last_peered":"2024-04-25T11:20:59.924998+0000","last_clean":"2024-04-25T11:20:59.924998+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:59.924998+0000","last_undegraded":"2024-04-25T11:20:59.924998+0000","last_fullsized":"2024-04-25T11:20:59.924998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932179+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:59.932179+0000","last_peered":"2024-04-25T11:20:59.932179+0000","last_clean":"2024-04-25T11:20:59.932179+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:59.932179+0000","last_undegraded":"2024-04-25T11:20:59.932179+0000","last_fullsized":"2024-04-25T11:20:59.932179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474317+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:21:00.474317+0000","last_peered":"2024-04-25T11:21:00.474317+0000","last_clean":"2024-04-25T11:21:00.474317+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:21:00.474317+0000","last_undegraded":"2024-04-25T11:21:00.474317+0000","last_fullsized":"2024-04-25T11:21:00.474317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929163+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:59.929163+0000","last_peered":"2024-04-25T11:20:59.929163+0000","last_clean":"2024-04-25T11:20:59.929163+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:59.929163+0000","last_undegraded":"2024-04-25T11:20:59.929163+0000","last_fullsized":"2024-04-25T11:20:59.929163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.475936+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:21:00.475936+0000","last_peered":"2024-04-25T11:21:00.475936+0000","last_clean":"2024-04-25T11:21:00.475936+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:21:00.475936+0000","last_undegraded":"2024-04-25T11:21:00.475936+0000","last_fullsized":"2024-04-25T11:21:00.475936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936290+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:59.936290+0000","last_peered":"2024-04-25T11:20:59.936290+0000","last_clean":"2024-04-25T11:20:59.936290+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:59.936290+0000","last_undegraded":"2024-04-25T11:20:59.936290+0000","last_fullsized":"2024-04-25T11:20:59.936290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924776+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:59.924776+0000","last_peered":"2024-04-25T11:20:59.924776+0000","last_clean":"2024-04-25T11:20:59.924776+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:59.924776+0000","last_undegraded":"2024-04-25T11:20:59.924776+0000","last_fullsized":"2024-04-25T11:20:59.924776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.945079+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:20:59.945079+0000","last_peered":"2024-04-25T11:20:59.945079+0000","last_clean":"2024-04-25T11:20:59.945079+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:20:59.945079+0000","last_undegraded":"2024-04-25T11:20:59.945079+0000","last_fullsized":"2024-04-25T11:20:59.945079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926127+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:59.926127+0000","last_peered":"2024-04-25T11:20:59.926127+0000","last_clean":"2024-04-25T11:20:59.926127+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:59.926127+0000","last_undegraded":"2024-04-25T11:20:59.926127+0000","last_fullsized":"2024-04-25T11:20:59.926127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.477014+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:21:00.477014+0000","last_peered":"2024-04-25T11:21:00.477014+0000","last_clean":"2024-04-25T11:21:00.477014+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:21:00.477014+0000","last_undegraded":"2024-04-25T11:21:00.477014+0000","last_fullsized":"2024-04-25T11:21:00.477014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474387+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474387+0000","last_peered":"2024-04-25T11:21:00.474387+0000","last_clean":"2024-04-25T11:21:00.474387+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:21:00.474387+0000","last_undegraded":"2024-04-25T11:21:00.474387+0000","last_fullsized":"2024-04-25T11:21:00.474387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933616+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:59.933616+0000","last_peered":"2024-04-25T11:20:59.933616+0000","last_clean":"2024-04-25T11:20:59.933616+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:59.933616+0000","last_undegraded":"2024-04-25T11:20:59.933616+0000","last_fullsized":"2024-04-25T11:20:59.933616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935312+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:59.935312+0000","last_peered":"2024-04-25T11:20:59.935312+0000","last_clean":"2024-04-25T11:20:59.935312+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:59.935312+0000","last_undegraded":"2024-04-25T11:20:59.935312+0000","last_fullsized":"2024-04-25T11:20:59.935312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932719+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:59.932719+0000","last_peered":"2024-04-25T11:20:59.932719+0000","last_clean":"2024-04-25T11:20:59.932719+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:59.932719+0000","last_undegraded":"2024-04-25T11:20:59.932719+0000","last_fullsized":"2024-04-25T11:20:59.932719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068007+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:21:00.068007+0000","last_peered":"2024-04-25T11:21:00.068007+0000","last_clean":"2024-04-25T11:21:00.068007+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:21:00.068007+0000","last_undegraded":"2024-04-25T11:21:00.068007+0000","last_fullsized":"2024-04-25T11:21:00.068007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925200+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:59.925200+0000","last_peered":"2024-04-25T11:20:59.925200+0000","last_clean":"2024-04-25T11:20:59.925200+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:59.925200+0000","last_undegraded":"2024-04-25T11:20:59.925200+0000","last_fullsized":"2024-04-25T11:20:59.925200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924837+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:59.924837+0000","last_peered":"2024-04-25T11:20:59.924837+0000","last_clean":"2024-04-25T11:20:59.924837+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:59.924837+0000","last_undegraded":"2024-04-25T11:20:59.924837+0000","last_fullsized":"2024-04-25T11:20:59.924837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111746+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:21:00.111746+0000","last_peered":"2024-04-25T11:21:00.111746+0000","last_clean":"2024-04-25T11:21:00.111746+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:21:00.111746+0000","last_undegraded":"2024-04-25T11:21:00.111746+0000","last_fullsized":"2024-04-25T11:21:00.111746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926148+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:59.926148+0000","last_peered":"2024-04-25T11:20:59.926148+0000","last_clean":"2024-04-25T11:20:59.926148+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:59.926148+0000","last_undegraded":"2024-04-25T11:20:59.926148+0000","last_fullsized":"2024-04-25T11:20:59.926148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109471+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:21:00.109471+0000","last_peered":"2024-04-25T11:21:00.109471+0000","last_clean":"2024-04-25T11:21:00.109471+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:21:00.109471+0000","last_undegraded":"2024-04-25T11:21:00.109471+0000","last_fullsized":"2024-04-25T11:21:00.109471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474518+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:21:00.474518+0000","last_peered":"2024-04-25T11:21:00.474518+0000","last_clean":"2024-04-25T11:21:00.474518+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:21:00.474518+0000","last_undegraded":"2024-04-25T11:21:00.474518+0000","last_fullsized":"2024-04-25T11:21:00.474518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109512+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:21:00.109512+0000","last_peered":"2024-04-25T11:21:00.109512+0000","last_clean":"2024-04-25T11:21:00.109512+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:21:00.109512+0000","last_undegraded":"2024-04-25T11:21:00.109512+0000","last_fullsized":"2024-04-25T11:21:00.109512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474969+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:21:00.474969+0000","last_peered":"2024-04-25T11:21:00.474969+0000","last_clean":"2024-04-25T11:21:00.474969+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:21:00.474969+0000","last_undegraded":"2024-04-25T11:21:00.474969+0000","last_fullsized":"2024-04-25T11:21:00.474969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938661+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:59.938661+0000","last_peered":"2024-04-25T11:20:59.938661+0000","last_clean":"2024-04-25T11:20:59.938661+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:59.938661+0000","last_undegraded":"2024-04-25T11:20:59.938661+0000","last_fullsized":"2024-04-25T11:20:59.938661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939739+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:59.939739+0000","last_peered":"2024-04-25T11:20:59.939739+0000","last_clean":"2024-04-25T11:20:59.939739+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:59.939739+0000","last_undegraded":"2024-04-25T11:20:59.939739+0000","last_fullsized":"2024-04-25T11:20:59.939739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109891+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:21:00.109891+0000","last_peered":"2024-04-25T11:21:00.109891+0000","last_clean":"2024-04-25T11:21:00.109891+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:21:00.109891+0000","last_undegraded":"2024-04-25T11:21:00.109891+0000","last_fullsized":"2024-04-25T11:21:00.109891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12028,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":707270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":119875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":118734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":116132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":119909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":116264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":115893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":901120,"data_stored":705603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:21:27.021 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-25T11:21:27.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:21:27.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.207+0000 7f1622cba640 1 Processor -- start 2024-04-25T11:21:27.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.207+0000 7f1622cba640 1 -- start start 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1622cba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f161c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1622cba640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f161c160370 0x7f161c0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1622cba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f161c15c6f0 con 0x7f161c15c0d0 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1622cba640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f161c0c6310 con 0x7f161c160370 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1622cba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f161c15c990 con 0x7f161c15ff60 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1620a56640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f161c15c0d0 0x7f161c0cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48118/0 (socket says 172.21.15.136:48118) 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1620a56640 1 -- 172.21.15.136:0/1824352832 learned_addr learned my addr 172.21.15.136:0/1824352832 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161bfff640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f161c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205201136 0 0) 0x7f161c15c6f0 con 0x7f161c15c0d0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 --2- 172.21.15.136:0/1824352832 >> v2:172.21.15.157:3300/0 conn(0x7f161c160370 0x7f161c0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1608003530 con 0x7f161c15c0d0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3183812453 0 0) 0x7f1608003530 con 0x7f161c15c0d0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 msgr2=0x7f161c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f161c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 >> v2:172.21.15.157:3300/0 conn(0x7f161c160370 msgr2=0x7f161c0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 --2- 172.21.15.136:0/1824352832 >> v2:172.21.15.157:3300/0 conn(0x7f161c160370 0x7f161c0ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c15cc30 con 0x7f161c15c0d0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f1614001680 con 0x7f161c15c0d0 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 >> v1:172.21.15.136:6789/0 conn(0x7f161c15c0d0 legacy=0x7f161c0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f16080039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1608003f40 0x7f1608004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f1608005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f161c0c6310 con 0x7f1608004d50 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f161c15c990 con 0x7f1608003f40 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f160800bad0 con 0x7f161c15ff60 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f1608005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1620a56640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f16080039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161bfff640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1608003f40 0x7f1608004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 -- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 msgr2=0x7f16080039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f16080039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 -- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1608003f40 msgr2=0x7f1608004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1608003f40 0x7f1608004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1621257640 1 -- 172.21.15.136:0/1824352832 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1608003530 con 0x7f1608004d50 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f161bfff640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1608003f40 0x7f1608004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:27.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.211+0000 7f1620a56640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c15ff60 0x7f16080039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1621257640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f1608005160 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1604009ab0 tx=0x7f160401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7468fc2adf64a515 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040b2070 con 0x7f1608004d50 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16040a5640 con 0x7f1608004d50 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f161b7fe640 1 -- 172.21.15.136:0/1824352832 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040096b0 con 0x7f1608004d50 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- 172.21.15.136:0/1824352832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 msgr2=0x7f1608005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 --2- 172.21.15.136:0/1824352832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f1608005160 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1604009ab0 tx=0x7f160401ca50 comp rx=0 tx=0).stop 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1621257640 1 -- 172.21.15.136:0/1824352832 reap_dead start 2024-04-25T11:21:27.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- 172.21.15.136:0/1824352832 shutdown_connections 2024-04-25T11:21:27.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- 172.21.15.136:0/1824352832 >> 172.21.15.136:0/1824352832 conn(0x7f161c06f9d0 msgr2=0x7f161c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- 172.21.15.136:0/1824352832 shutdown_connections 2024-04-25T11:21:27.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- 172.21.15.136:0/1824352832 wait complete. 2024-04-25T11:21:27.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 Processor -- start 2024-04-25T11:21:27.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.215+0000 7f1622cba640 1 -- start start 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c1538d0 0x7f161c0719c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 0x7f161c1f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f161c15cf10 con 0x7f1608004d50 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f161c15fa40 con 0x7f161c14fdf0 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f161c0c9170 con 0x7f161c1538d0 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37100/0 (socket says 172.21.15.136:37100) 2024-04-25T11:21:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 -- 172.21.15.136:0/90269255 learned_addr learned my addr 172.21.15.136:0/90269255 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f161bfff640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c1538d0 0x7f161c0719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1621257640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 0x7f161c1f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 -- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c1538d0 msgr2=0x7f161c0719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c1538d0 0x7f161c0719c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 -- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 msgr2=0x7f161c1f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 0x7f161c1f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 -- 172.21.15.136:0/90269255 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c1f4270 con 0x7f1608004d50 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1621257640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 0x7f161c1f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1620a56640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f161400d8e0 tx=0x7f161400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614019070 con 0x7f1608004d50 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16140074e0 con 0x7f1608004d50 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c0cc1e0 con 0x7f1608004d50 2024-04-25T11:21:27.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614004a10 con 0x7f1608004d50 2024-04-25T11:21:27.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c0cca60 con 0x7f1608004d50 2024-04-25T11:21:27.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.219+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f161c06a350 con 0x7f1608004d50 2024-04-25T11:21:27.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.223+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f161401c070 con 0x7f1608004d50 2024-04-25T11:21:27.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.223+0000 7f16197fa640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 0x7f15fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.223+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f161408f700 con 0x7f1608004d50 2024-04-25T11:21:27.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.223+0000 7f161bfff640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 0x7f15fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.223+0000 7f161bfff640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 0x7f15fc071910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f161c072360 tx=0x7f160c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.227+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1614023020 con 0x7f1608004d50 2024-04-25T11:21:27.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.383+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f161c14c230 con 0x7f15fc06f400 2024-04-25T11:21:27.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.407+0000 7f16197fa640 1 -- 172.21.15.136:0/90269255 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332648 (secure 0 0 0) 0x7f161c14c230 con 0x7f15fc06f400 2024-04-25T11:21:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:27.412 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:21:27.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 msgr2=0x7f15fc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 0x7f15fc071910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f161c072360 tx=0x7f160c00b040 comp rx=0 tx=0).stop 2024-04-25T11:21:27.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 msgr2=0x7f161c1f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f161400d8e0 tx=0x7f161400dda0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 shutdown_connections 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f161c14fdf0 0x7f161c1f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15fc06f400 0x7f15fc071910 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f161c1538d0 0x7f161c0719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 --2- 172.21.15.136:0/90269255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1608004d50 0x7f161c1f3d30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 >> 172.21.15.136:0/90269255 conn(0x7f161c06f9d0 msgr2=0x7f161c0c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 shutdown_connections 2024-04-25T11:21:27.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.415+0000 7f1622cba640 1 -- 172.21.15.136:0/90269255 wait complete. 2024-04-25T11:21:27.431 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-25T11:21:26.779999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602396,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":317,"num_write_kb":1702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236360,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738360,"statfs":{"total":773094113280,"available":772852080640,"internally_reserved":0,"allocated":3678208,"data_stored":2119680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3170,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005169"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.994195+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:21:11.994195+0000","last_peered":"2024-04-25T11:21:11.994195+0000","last_clean":"2024-04-25T11:21:11.994195+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:21:11.994195+0000","last_undegraded":"2024-04-25T11:21:11.994195+0000","last_fullsized":"2024-04-25T11:21:11.994195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931730+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:20:59.931730+0000","last_peered":"2024-04-25T11:20:59.931730+0000","last_clean":"2024-04-25T11:20:59.931730+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:20:59.931730+0000","last_undegraded":"2024-04-25T11:20:59.931730+0000","last_fullsized":"2024-04-25T11:20:59.931730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928414+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:20:59.928414+0000","last_peered":"2024-04-25T11:20:59.928414+0000","last_clean":"2024-04-25T11:20:59.928414+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:20:59.928414+0000","last_undegraded":"2024-04-25T11:20:59.928414+0000","last_fullsized":"2024-04-25T11:20:59.928414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932201+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:20:59.932201+0000","last_peered":"2024-04-25T11:20:59.932201+0000","last_clean":"2024-04-25T11:20:59.932201+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:20:59.932201+0000","last_undegraded":"2024-04-25T11:20:59.932201+0000","last_fullsized":"2024-04-25T11:20:59.932201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940417+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:20:59.940417+0000","last_peered":"2024-04-25T11:20:59.940417+0000","last_clean":"2024-04-25T11:20:59.940417+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:20:59.940417+0000","last_undegraded":"2024-04-25T11:20:59.940417+0000","last_fullsized":"2024-04-25T11:20:59.940417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068980+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:21:00.068980+0000","last_peered":"2024-04-25T11:21:00.068980+0000","last_clean":"2024-04-25T11:21:00.068980+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:21:00.068980+0000","last_undegraded":"2024-04-25T11:21:00.068980+0000","last_fullsized":"2024-04-25T11:21:00.068980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.003480+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:21:12.003480+0000","last_peered":"2024-04-25T11:21:12.003480+0000","last_clean":"2024-04-25T11:21:12.003480+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:12.003480+0000","last_undegraded":"2024-04-25T11:21:12.003480+0000","last_fullsized":"2024-04-25T11:21:12.003480+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075805+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:21:00.075805+0000","last_peered":"2024-04-25T11:21:00.075805+0000","last_clean":"2024-04-25T11:21:00.075805+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:21:00.075805+0000","last_undegraded":"2024-04-25T11:21:00.075805+0000","last_fullsized":"2024-04-25T11:21:00.075805+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069307+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:21:00.069307+0000","last_peered":"2024-04-25T11:21:00.069307+0000","last_clean":"2024-04-25T11:21:00.069307+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:21:00.069307+0000","last_undegraded":"2024-04-25T11:21:00.069307+0000","last_fullsized":"2024-04-25T11:21:00.069307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002064+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:21:12.002064+0000","last_peered":"2024-04-25T11:21:12.002064+0000","last_clean":"2024-04-25T11:21:12.002064+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:21:12.002064+0000","last_undegraded":"2024-04-25T11:21:12.002064+0000","last_fullsized":"2024-04-25T11:21:12.002064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.075553+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:21:00.075553+0000","last_peered":"2024-04-25T11:21:00.075553+0000","last_clean":"2024-04-25T11:21:00.075553+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:21:00.075553+0000","last_undegraded":"2024-04-25T11:21:00.075553+0000","last_fullsized":"2024-04-25T11:21:00.075553+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069045+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:21:00.069045+0000","last_peered":"2024-04-25T11:21:00.069045+0000","last_clean":"2024-04-25T11:21:00.069045+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:21:00.069045+0000","last_undegraded":"2024-04-25T11:21:00.069045+0000","last_fullsized":"2024-04-25T11:21:00.069045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924874+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:20:59.924874+0000","last_peered":"2024-04-25T11:20:59.924874+0000","last_clean":"2024-04-25T11:20:59.924874+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:20:59.924874+0000","last_undegraded":"2024-04-25T11:20:59.924874+0000","last_fullsized":"2024-04-25T11:20:59.924874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111184+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:21:00.111184+0000","last_peered":"2024-04-25T11:21:00.111184+0000","last_clean":"2024-04-25T11:21:00.111184+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:21:00.111184+0000","last_undegraded":"2024-04-25T11:21:00.111184+0000","last_fullsized":"2024-04-25T11:21:00.111184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070128+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:21:00.070128+0000","last_peered":"2024-04-25T11:21:00.070128+0000","last_clean":"2024-04-25T11:21:00.070128+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:21:00.070128+0000","last_undegraded":"2024-04-25T11:21:00.070128+0000","last_fullsized":"2024-04-25T11:21:00.070128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930588+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:20:59.930588+0000","last_peered":"2024-04-25T11:20:59.930588+0000","last_clean":"2024-04-25T11:20:59.930588+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:20:59.930588+0000","last_undegraded":"2024-04-25T11:20:59.930588+0000","last_fullsized":"2024-04-25T11:20:59.930588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959714+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:21:05.959714+0000","last_peered":"2024-04-25T11:21:05.959714+0000","last_clean":"2024-04-25T11:21:05.959714+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:21:05.959714+0000","last_undegraded":"2024-04-25T11:21:05.959714+0000","last_fullsized":"2024-04-25T11:21:05.959714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069432+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:21:00.069432+0000","last_peered":"2024-04-25T11:21:00.069432+0000","last_clean":"2024-04-25T11:21:00.069432+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:21:00.069432+0000","last_undegraded":"2024-04-25T11:21:00.069432+0000","last_fullsized":"2024-04-25T11:21:00.069432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926697+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:20:59.926697+0000","last_peered":"2024-04-25T11:20:59.926697+0000","last_clean":"2024-04-25T11:20:59.926697+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:20:59.926697+0000","last_undegraded":"2024-04-25T11:20:59.926697+0000","last_fullsized":"2024-04-25T11:20:59.926697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.991983+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:21:11.991983+0000","last_peered":"2024-04-25T11:21:11.991983+0000","last_clean":"2024-04-25T11:21:11.991983+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:21:11.991983+0000","last_undegraded":"2024-04-25T11:21:11.991983+0000","last_fullsized":"2024-04-25T11:21:11.991983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.114813+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:21:00.114813+0000","last_peered":"2024-04-25T11:21:00.114813+0000","last_clean":"2024-04-25T11:21:00.114813+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:21:00.114813+0000","last_undegraded":"2024-04-25T11:21:00.114813+0000","last_fullsized":"2024-04-25T11:21:00.114813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067738+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:21:00.067738+0000","last_peered":"2024-04-25T11:21:00.067738+0000","last_clean":"2024-04-25T11:21:00.067738+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:21:00.067738+0000","last_undegraded":"2024-04-25T11:21:00.067738+0000","last_fullsized":"2024-04-25T11:21:00.067738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110139+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:21:00.110139+0000","last_peered":"2024-04-25T11:21:00.110139+0000","last_clean":"2024-04-25T11:21:00.110139+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:21:00.110139+0000","last_undegraded":"2024-04-25T11:21:00.110139+0000","last_fullsized":"2024-04-25T11:21:00.110139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070453+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:21:00.070453+0000","last_peered":"2024-04-25T11:21:00.070453+0000","last_clean":"2024-04-25T11:21:00.070453+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:21:00.070453+0000","last_undegraded":"2024-04-25T11:21:00.070453+0000","last_fullsized":"2024-04-25T11:21:00.070453+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940092+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:20:59.940092+0000","last_peered":"2024-04-25T11:20:59.940092+0000","last_clean":"2024-04-25T11:20:59.940092+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:20:59.940092+0000","last_undegraded":"2024-04-25T11:20:59.940092+0000","last_fullsized":"2024-04-25T11:20:59.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:12.002572+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:21:12.002572+0000","last_peered":"2024-04-25T11:21:12.002572+0000","last_clean":"2024-04-25T11:21:12.002572+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:21:12.002572+0000","last_undegraded":"2024-04-25T11:21:12.002572+0000","last_fullsized":"2024-04-25T11:21:12.002572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069767+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:21:00.069767+0000","last_peered":"2024-04-25T11:21:00.069767+0000","last_clean":"2024-04-25T11:21:00.069767+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:21:00.069767+0000","last_undegraded":"2024-04-25T11:21:00.069767+0000","last_fullsized":"2024-04-25T11:21:00.069767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932437+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:20:59.932437+0000","last_peered":"2024-04-25T11:20:59.932437+0000","last_clean":"2024-04-25T11:20:59.932437+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:20:59.932437+0000","last_undegraded":"2024-04-25T11:20:59.932437+0000","last_fullsized":"2024-04-25T11:20:59.932437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962037+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:21:05.962037+0000","last_peered":"2024-04-25T11:21:05.962037+0000","last_clean":"2024-04-25T11:21:05.962037+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:21:05.962037+0000","last_undegraded":"2024-04-25T11:21:05.962037+0000","last_fullsized":"2024-04-25T11:21:05.962037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925818+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:20:59.925818+0000","last_peered":"2024-04-25T11:20:59.925818+0000","last_clean":"2024-04-25T11:20:59.925818+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:20:59.925818+0000","last_undegraded":"2024-04-25T11:20:59.925818+0000","last_fullsized":"2024-04-25T11:20:59.925818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936496+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:20:59.936496+0000","last_peered":"2024-04-25T11:20:59.936496+0000","last_clean":"2024-04-25T11:20:59.936496+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:20:59.936496+0000","last_undegraded":"2024-04-25T11:20:59.936496+0000","last_fullsized":"2024-04-25T11:20:59.936496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.962135+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:21:05.962135+0000","last_peered":"2024-04-25T11:21:05.962135+0000","last_clean":"2024-04-25T11:21:05.962135+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:21:05.962135+0000","last_undegraded":"2024-04-25T11:21:05.962135+0000","last_fullsized":"2024-04-25T11:21:05.962135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935995+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:20:59.935995+0000","last_peered":"2024-04-25T11:20:59.935995+0000","last_clean":"2024-04-25T11:20:59.935995+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:20:59.935995+0000","last_undegraded":"2024-04-25T11:20:59.935995+0000","last_fullsized":"2024-04-25T11:20:59.935995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474622+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474622+0000","last_peered":"2024-04-25T11:21:00.474622+0000","last_clean":"2024-04-25T11:21:00.474622+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:21:00.474622+0000","last_undegraded":"2024-04-25T11:21:00.474622+0000","last_fullsized":"2024-04-25T11:21:00.474622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"26'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936918+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:20:59.936918+0000","last_peered":"2024-04-25T11:20:59.936918+0000","last_clean":"2024-04-25T11:20:59.936918+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:20:59.936918+0000","last_undegraded":"2024-04-25T11:20:59.936918+0000","last_fullsized":"2024-04-25T11:20:59.936918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936370+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:20:59.936370+0000","last_peered":"2024-04-25T11:20:59.936370+0000","last_clean":"2024-04-25T11:20:59.936370+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:20:59.936370+0000","last_undegraded":"2024-04-25T11:20:59.936370+0000","last_fullsized":"2024-04-25T11:20:59.936370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068094+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:21:00.068094+0000","last_peered":"2024-04-25T11:21:00.068094+0000","last_clean":"2024-04-25T11:21:00.068094+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:21:00.068094+0000","last_undegraded":"2024-04-25T11:21:00.068094+0000","last_fullsized":"2024-04-25T11:21:00.068094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.999818+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:21:11.999818+0000","last_peered":"2024-04-25T11:21:11.999818+0000","last_clean":"2024-04-25T11:21:11.999818+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:21:11.999818+0000","last_undegraded":"2024-04-25T11:21:11.999818+0000","last_fullsized":"2024-04-25T11:21:11.999818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:20:59.936416+0000","last_peered":"2024-04-25T11:20:59.936416+0000","last_clean":"2024-04-25T11:20:59.936416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:20:59.936416+0000","last_undegraded":"2024-04-25T11:20:59.936416+0000","last_fullsized":"2024-04-25T11:20:59.936416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.934845+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:20:59.934845+0000","last_peered":"2024-04-25T11:20:59.934845+0000","last_clean":"2024-04-25T11:20:59.934845+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:20:59.934845+0000","last_undegraded":"2024-04-25T11:20:59.934845+0000","last_fullsized":"2024-04-25T11:20:59.934845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992803+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:21:11.992803+0000","last_peered":"2024-04-25T11:21:11.992803+0000","last_clean":"2024-04-25T11:21:11.992803+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:21:11.992803+0000","last_undegraded":"2024-04-25T11:21:11.992803+0000","last_fullsized":"2024-04-25T11:21:11.992803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939030+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:20:59.939030+0000","last_peered":"2024-04-25T11:20:59.939030+0000","last_clean":"2024-04-25T11:20:59.939030+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:20:59.939030+0000","last_undegraded":"2024-04-25T11:20:59.939030+0000","last_fullsized":"2024-04-25T11:20:59.939030+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933545+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:20:59.933545+0000","last_peered":"2024-04-25T11:20:59.933545+0000","last_clean":"2024-04-25T11:20:59.933545+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:20:59.933545+0000","last_undegraded":"2024-04-25T11:20:59.933545+0000","last_fullsized":"2024-04-25T11:20:59.933545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.917110+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:21:00.917110+0000","last_peered":"2024-04-25T11:21:00.917110+0000","last_clean":"2024-04-25T11:21:00.917110+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:21:00.917110+0000","last_undegraded":"2024-04-25T11:21:00.917110+0000","last_fullsized":"2024-04-25T11:21:00.917110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939772+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:20:59.939772+0000","last_peered":"2024-04-25T11:20:59.939772+0000","last_clean":"2024-04-25T11:20:59.939772+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:20:59.939772+0000","last_undegraded":"2024-04-25T11:20:59.939772+0000","last_fullsized":"2024-04-25T11:20:59.939772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929859+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:20:59.929859+0000","last_peered":"2024-04-25T11:20:59.929859+0000","last_clean":"2024-04-25T11:20:59.929859+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:20:59.929859+0000","last_undegraded":"2024-04-25T11:20:59.929859+0000","last_fullsized":"2024-04-25T11:20:59.929859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960397+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:21:05.960397+0000","last_peered":"2024-04-25T11:21:05.960397+0000","last_clean":"2024-04-25T11:21:05.960397+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:21:05.960397+0000","last_undegraded":"2024-04-25T11:21:05.960397+0000","last_fullsized":"2024-04-25T11:21:05.960397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":122,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110312+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:21:00.110312+0000","last_peered":"2024-04-25T11:21:00.110312+0000","last_clean":"2024-04-25T11:21:00.110312+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:21:00.110312+0000","last_undegraded":"2024-04-25T11:21:00.110312+0000","last_fullsized":"2024-04-25T11:21:00.110312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924917+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:20:59.924917+0000","last_peered":"2024-04-25T11:20:59.924917+0000","last_clean":"2024-04-25T11:20:59.924917+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:20:59.924917+0000","last_undegraded":"2024-04-25T11:20:59.924917+0000","last_fullsized":"2024-04-25T11:20:59.924917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111023+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:21:00.111023+0000","last_peered":"2024-04-25T11:21:00.111023+0000","last_clean":"2024-04-25T11:21:00.111023+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:21:00.111023+0000","last_undegraded":"2024-04-25T11:21:00.111023+0000","last_fullsized":"2024-04-25T11:21:00.111023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961234+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:21:05.961234+0000","last_peered":"2024-04-25T11:21:05.961234+0000","last_clean":"2024-04-25T11:21:05.961234+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:21:05.961234+0000","last_undegraded":"2024-04-25T11:21:05.961234+0000","last_fullsized":"2024-04-25T11:21:05.961234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924269+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:20:59.924269+0000","last_peered":"2024-04-25T11:20:59.924269+0000","last_clean":"2024-04-25T11:20:59.924269+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:20:59.924269+0000","last_undegraded":"2024-04-25T11:20:59.924269+0000","last_fullsized":"2024-04-25T11:20:59.924269+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474615+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:21:00.474615+0000","last_peered":"2024-04-25T11:21:00.474615+0000","last_clean":"2024-04-25T11:21:00.474615+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:21:00.474615+0000","last_undegraded":"2024-04-25T11:21:00.474615+0000","last_fullsized":"2024-04-25T11:21:00.474615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961055+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:21:05.961055+0000","last_peered":"2024-04-25T11:21:05.961055+0000","last_clean":"2024-04-25T11:21:05.961055+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:21:05.961055+0000","last_undegraded":"2024-04-25T11:21:05.961055+0000","last_fullsized":"2024-04-25T11:21:05.961055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.479066+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:21:00.479066+0000","last_peered":"2024-04-25T11:21:00.479066+0000","last_clean":"2024-04-25T11:21:00.479066+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:21:00.479066+0000","last_undegraded":"2024-04-25T11:21:00.479066+0000","last_fullsized":"2024-04-25T11:21:00.479066+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111834+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:21:00.111834+0000","last_peered":"2024-04-25T11:21:00.111834+0000","last_clean":"2024-04-25T11:21:00.111834+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:21:00.111834+0000","last_undegraded":"2024-04-25T11:21:00.111834+0000","last_fullsized":"2024-04-25T11:21:00.111834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992924+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:21:11.992924+0000","last_peered":"2024-04-25T11:21:11.992924+0000","last_clean":"2024-04-25T11:21:11.992924+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:21:11.992924+0000","last_undegraded":"2024-04-25T11:21:11.992924+0000","last_fullsized":"2024-04-25T11:21:11.992924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474787+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:21:00.474787+0000","last_peered":"2024-04-25T11:21:00.474787+0000","last_clean":"2024-04-25T11:21:00.474787+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:21:00.474787+0000","last_undegraded":"2024-04-25T11:21:00.474787+0000","last_fullsized":"2024-04-25T11:21:00.474787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111717+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:21:00.111717+0000","last_peered":"2024-04-25T11:21:00.111717+0000","last_clean":"2024-04-25T11:21:00.111717+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:21:00.111717+0000","last_undegraded":"2024-04-25T11:21:00.111717+0000","last_fullsized":"2024-04-25T11:21:00.111717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'2","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070606+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:21:00.070606+0000","last_peered":"2024-04-25T11:21:00.070606+0000","last_clean":"2024-04-25T11:21:00.070606+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:21:00.070606+0000","last_undegraded":"2024-04-25T11:21:00.070606+0000","last_fullsized":"2024-04-25T11:21:00.070606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069701+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:21:00.069701+0000","last_peered":"2024-04-25T11:21:00.069701+0000","last_clean":"2024-04-25T11:21:00.069701+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:21:00.069701+0000","last_undegraded":"2024-04-25T11:21:00.069701+0000","last_fullsized":"2024-04-25T11:21:00.069701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110638+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:21:00.110638+0000","last_peered":"2024-04-25T11:21:00.110638+0000","last_clean":"2024-04-25T11:21:00.110638+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:21:00.110638+0000","last_undegraded":"2024-04-25T11:21:00.110638+0000","last_fullsized":"2024-04-25T11:21:00.110638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.957738+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:21:05.957738+0000","last_peered":"2024-04-25T11:21:05.957738+0000","last_clean":"2024-04-25T11:21:05.957738+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:21:05.957738+0000","last_undegraded":"2024-04-25T11:21:05.957738+0000","last_fullsized":"2024-04-25T11:21:05.957738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939656+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:20:59.939656+0000","last_peered":"2024-04-25T11:20:59.939656+0000","last_clean":"2024-04-25T11:20:59.939656+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:20:59.939656+0000","last_undegraded":"2024-04-25T11:20:59.939656+0000","last_fullsized":"2024-04-25T11:20:59.939656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924486+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:20:59.924486+0000","last_peered":"2024-04-25T11:20:59.924486+0000","last_clean":"2024-04-25T11:20:59.924486+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:20:59.924486+0000","last_undegraded":"2024-04-25T11:20:59.924486+0000","last_fullsized":"2024-04-25T11:20:59.924486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961276+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:21:05.961276+0000","last_peered":"2024-04-25T11:21:05.961276+0000","last_clean":"2024-04-25T11:21:05.961276+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:21:05.961276+0000","last_undegraded":"2024-04-25T11:21:05.961276+0000","last_fullsized":"2024-04-25T11:21:05.961276+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931983+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:20:59.931983+0000","last_peered":"2024-04-25T11:20:59.931983+0000","last_clean":"2024-04-25T11:20:59.931983+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:20:59.931983+0000","last_undegraded":"2024-04-25T11:20:59.931983+0000","last_fullsized":"2024-04-25T11:20:59.931983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.110991+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:21:00.110991+0000","last_peered":"2024-04-25T11:21:00.110991+0000","last_clean":"2024-04-25T11:21:00.110991+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:21:00.110991+0000","last_undegraded":"2024-04-25T11:21:00.110991+0000","last_fullsized":"2024-04-25T11:21:00.110991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.959522+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:21:05.959522+0000","last_peered":"2024-04-25T11:21:05.959522+0000","last_clean":"2024-04-25T11:21:05.959522+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:21:05.959522+0000","last_undegraded":"2024-04-25T11:21:05.959522+0000","last_fullsized":"2024-04-25T11:21:05.959522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.930980+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:20:59.930980+0000","last_peered":"2024-04-25T11:20:59.930980+0000","last_clean":"2024-04-25T11:20:59.930980+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:20:59.930980+0000","last_undegraded":"2024-04-25T11:20:59.930980+0000","last_fullsized":"2024-04-25T11:20:59.930980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935426+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:20:59.935426+0000","last_peered":"2024-04-25T11:20:59.935426+0000","last_clean":"2024-04-25T11:20:59.935426+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:20:59.935426+0000","last_undegraded":"2024-04-25T11:20:59.935426+0000","last_fullsized":"2024-04-25T11:20:59.935426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":34,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.992627+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:21:11.992627+0000","last_peered":"2024-04-25T11:21:11.992627+0000","last_clean":"2024-04-25T11:21:11.992627+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:21:11.992627+0000","last_undegraded":"2024-04-25T11:21:11.992627+0000","last_fullsized":"2024-04-25T11:21:11.992627+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476378+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:21:00.476378+0000","last_peered":"2024-04-25T11:21:00.476378+0000","last_clean":"2024-04-25T11:21:00.476378+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:21:00.476378+0000","last_undegraded":"2024-04-25T11:21:00.476378+0000","last_fullsized":"2024-04-25T11:21:00.476378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109709+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:21:00.109709+0000","last_peered":"2024-04-25T11:21:00.109709+0000","last_clean":"2024-04-25T11:21:00.109709+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:21:00.109709+0000","last_undegraded":"2024-04-25T11:21:00.109709+0000","last_fullsized":"2024-04-25T11:21:00.109709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'16","reported_seq":42,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:20.836111+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:21:20.836111+0000","last_peered":"2024-04-25T11:21:20.836111+0000","last_clean":"2024-04-25T11:21:20.836111+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:21:20.836111+0000","last_undegraded":"2024-04-25T11:21:20.836111+0000","last_fullsized":"2024-04-25T11:21:20.836111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4283,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936662+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:20:59.936662+0000","last_peered":"2024-04-25T11:20:59.936662+0000","last_clean":"2024-04-25T11:20:59.936662+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:20:59.936662+0000","last_undegraded":"2024-04-25T11:20:59.936662+0000","last_fullsized":"2024-04-25T11:20:59.936662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.070136+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:21:00.070136+0000","last_peered":"2024-04-25T11:21:00.070136+0000","last_clean":"2024-04-25T11:21:00.070136+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:21:00.070136+0000","last_undegraded":"2024-04-25T11:21:00.070136+0000","last_fullsized":"2024-04-25T11:21:00.070136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938676+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:20:59.938676+0000","last_peered":"2024-04-25T11:20:59.938676+0000","last_clean":"2024-04-25T11:20:59.938676+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:20:59.938676+0000","last_undegraded":"2024-04-25T11:20:59.938676+0000","last_fullsized":"2024-04-25T11:20:59.938676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.069494+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:21:00.069494+0000","last_peered":"2024-04-25T11:21:00.069494+0000","last_clean":"2024-04-25T11:21:00.069494+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:21:00.069494+0000","last_undegraded":"2024-04-25T11:21:00.069494+0000","last_fullsized":"2024-04-25T11:21:00.069494+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.940805+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:20:59.940805+0000","last_peered":"2024-04-25T11:20:59.940805+0000","last_clean":"2024-04-25T11:20:59.940805+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:20:59.940805+0000","last_undegraded":"2024-04-25T11:20:59.940805+0000","last_fullsized":"2024-04-25T11:20:59.940805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:11.997669+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:21:11.997669+0000","last_peered":"2024-04-25T11:21:11.997669+0000","last_clean":"2024-04-25T11:21:11.997669+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:21:11.997669+0000","last_undegraded":"2024-04-25T11:21:11.997669+0000","last_fullsized":"2024-04-25T11:21:11.997669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.073978+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:21:00.073978+0000","last_peered":"2024-04-25T11:21:00.073978+0000","last_clean":"2024-04-25T11:21:00.073978+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:21:00.073978+0000","last_undegraded":"2024-04-25T11:21:00.073978+0000","last_fullsized":"2024-04-25T11:21:00.073978+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.476734+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:21:00.476734+0000","last_peered":"2024-04-25T11:21:00.476734+0000","last_clean":"2024-04-25T11:21:00.476734+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:21:00.476734+0000","last_undegraded":"2024-04-25T11:21:00.476734+0000","last_fullsized":"2024-04-25T11:21:00.476734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.960966+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:21:05.960966+0000","last_peered":"2024-04-25T11:21:05.960966+0000","last_clean":"2024-04-25T11:21:05.960966+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:21:05.960966+0000","last_undegraded":"2024-04-25T11:21:05.960966+0000","last_fullsized":"2024-04-25T11:21:05.960966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.928485+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:20:59.928485+0000","last_peered":"2024-04-25T11:20:59.928485+0000","last_clean":"2024-04-25T11:20:59.928485+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:20:59.928485+0000","last_undegraded":"2024-04-25T11:20:59.928485+0000","last_fullsized":"2024-04-25T11:20:59.928485+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938424+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:20:59.938424+0000","last_peered":"2024-04-25T11:20:59.938424+0000","last_clean":"2024-04-25T11:20:59.938424+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:20:59.938424+0000","last_undegraded":"2024-04-25T11:20:59.938424+0000","last_fullsized":"2024-04-25T11:20:59.938424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.944101+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:20:59.944101+0000","last_peered":"2024-04-25T11:20:59.944101+0000","last_clean":"2024-04-25T11:20:59.944101+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:20:59.944101+0000","last_undegraded":"2024-04-25T11:20:59.944101+0000","last_fullsized":"2024-04-25T11:20:59.944101+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.958897+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:21:05.958897+0000","last_peered":"2024-04-25T11:21:05.958897+0000","last_clean":"2024-04-25T11:21:05.958897+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:21:05.958897+0000","last_undegraded":"2024-04-25T11:21:05.958897+0000","last_fullsized":"2024-04-25T11:21:05.958897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924911+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:20:59.924911+0000","last_peered":"2024-04-25T11:20:59.924911+0000","last_clean":"2024-04-25T11:20:59.924911+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:20:59.924911+0000","last_undegraded":"2024-04-25T11:20:59.924911+0000","last_fullsized":"2024-04-25T11:20:59.924911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926423+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:20:59.926423+0000","last_peered":"2024-04-25T11:20:59.926423+0000","last_clean":"2024-04-25T11:20:59.926423+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:20:59.926423+0000","last_undegraded":"2024-04-25T11:20:59.926423+0000","last_fullsized":"2024-04-25T11:20:59.926423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.911858+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:21:00.911858+0000","last_peered":"2024-04-25T11:21:00.911858+0000","last_clean":"2024-04-25T11:21:00.911858+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:21:00.911858+0000","last_undegraded":"2024-04-25T11:21:00.911858+0000","last_fullsized":"2024-04-25T11:21:00.911858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:05.961666+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:21:05.961666+0000","last_peered":"2024-04-25T11:21:05.961666+0000","last_clean":"2024-04-25T11:21:05.961666+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:21:05.961666+0000","last_undegraded":"2024-04-25T11:21:05.961666+0000","last_fullsized":"2024-04-25T11:21:05.961666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936830+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:20:59.936830+0000","last_peered":"2024-04-25T11:20:59.936830+0000","last_clean":"2024-04-25T11:20:59.936830+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:20:59.936830+0000","last_undegraded":"2024-04-25T11:20:59.936830+0000","last_fullsized":"2024-04-25T11:20:59.936830+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924523+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:20:59.924523+0000","last_peered":"2024-04-25T11:20:59.924523+0000","last_clean":"2024-04-25T11:20:59.924523+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:20:59.924523+0000","last_undegraded":"2024-04-25T11:20:59.924523+0000","last_fullsized":"2024-04-25T11:20:59.924523+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"26'6","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931078+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:20:59.931078+0000","last_peered":"2024-04-25T11:20:59.931078+0000","last_clean":"2024-04-25T11:20:59.931078+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:20:59.931078+0000","last_undegraded":"2024-04-25T11:20:59.931078+0000","last_fullsized":"2024-04-25T11:20:59.931078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.067891+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:21:00.067891+0000","last_peered":"2024-04-25T11:21:00.067891+0000","last_clean":"2024-04-25T11:21:00.067891+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:21:00.067891+0000","last_undegraded":"2024-04-25T11:21:00.067891+0000","last_fullsized":"2024-04-25T11:21:00.067891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931524+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:20:59.931524+0000","last_peered":"2024-04-25T11:20:59.931524+0000","last_clean":"2024-04-25T11:20:59.931524+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:20:59.931524+0000","last_undegraded":"2024-04-25T11:20:59.931524+0000","last_fullsized":"2024-04-25T11:20:59.931524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924055+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:20:59.924055+0000","last_peered":"2024-04-25T11:20:59.924055+0000","last_clean":"2024-04-25T11:20:59.924055+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:20:59.924055+0000","last_undegraded":"2024-04-25T11:20:59.924055+0000","last_fullsized":"2024-04-25T11:20:59.924055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924236+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:20:59.924236+0000","last_peered":"2024-04-25T11:20:59.924236+0000","last_clean":"2024-04-25T11:20:59.924236+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:20:59.924236+0000","last_undegraded":"2024-04-25T11:20:59.924236+0000","last_fullsized":"2024-04-25T11:20:59.924236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474378+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:21:00.474378+0000","last_peered":"2024-04-25T11:21:00.474378+0000","last_clean":"2024-04-25T11:21:00.474378+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:21:00.474378+0000","last_undegraded":"2024-04-25T11:21:00.474378+0000","last_fullsized":"2024-04-25T11:21:00.474378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936557+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:20:59.936557+0000","last_peered":"2024-04-25T11:20:59.936557+0000","last_clean":"2024-04-25T11:20:59.936557+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:20:59.936557+0000","last_undegraded":"2024-04-25T11:20:59.936557+0000","last_fullsized":"2024-04-25T11:20:59.936557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.931409+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:20:59.931409+0000","last_peered":"2024-04-25T11:20:59.931409+0000","last_clean":"2024-04-25T11:20:59.931409+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:20:59.931409+0000","last_undegraded":"2024-04-25T11:20:59.931409+0000","last_fullsized":"2024-04-25T11:20:59.931409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938092+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:20:59.938092+0000","last_peered":"2024-04-25T11:20:59.938092+0000","last_clean":"2024-04-25T11:20:59.938092+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:20:59.938092+0000","last_undegraded":"2024-04-25T11:20:59.938092+0000","last_fullsized":"2024-04-25T11:20:59.938092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924998+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:20:59.924998+0000","last_peered":"2024-04-25T11:20:59.924998+0000","last_clean":"2024-04-25T11:20:59.924998+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:20:59.924998+0000","last_undegraded":"2024-04-25T11:20:59.924998+0000","last_fullsized":"2024-04-25T11:20:59.924998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932179+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:20:59.932179+0000","last_peered":"2024-04-25T11:20:59.932179+0000","last_clean":"2024-04-25T11:20:59.932179+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:20:59.932179+0000","last_undegraded":"2024-04-25T11:20:59.932179+0000","last_fullsized":"2024-04-25T11:20:59.932179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474317+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:21:00.474317+0000","last_peered":"2024-04-25T11:21:00.474317+0000","last_clean":"2024-04-25T11:21:00.474317+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:21:00.474317+0000","last_undegraded":"2024-04-25T11:21:00.474317+0000","last_fullsized":"2024-04-25T11:21:00.474317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.929163+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:20:59.929163+0000","last_peered":"2024-04-25T11:20:59.929163+0000","last_clean":"2024-04-25T11:20:59.929163+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:20:59.929163+0000","last_undegraded":"2024-04-25T11:20:59.929163+0000","last_fullsized":"2024-04-25T11:20:59.929163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.475936+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:21:00.475936+0000","last_peered":"2024-04-25T11:21:00.475936+0000","last_clean":"2024-04-25T11:21:00.475936+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:21:00.475936+0000","last_undegraded":"2024-04-25T11:21:00.475936+0000","last_fullsized":"2024-04-25T11:21:00.475936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.936290+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:20:59.936290+0000","last_peered":"2024-04-25T11:20:59.936290+0000","last_clean":"2024-04-25T11:20:59.936290+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:20:59.936290+0000","last_undegraded":"2024-04-25T11:20:59.936290+0000","last_fullsized":"2024-04-25T11:20:59.936290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924776+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:20:59.924776+0000","last_peered":"2024-04-25T11:20:59.924776+0000","last_clean":"2024-04-25T11:20:59.924776+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:20:59.924776+0000","last_undegraded":"2024-04-25T11:20:59.924776+0000","last_fullsized":"2024-04-25T11:20:59.924776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.945079+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:20:59.945079+0000","last_peered":"2024-04-25T11:20:59.945079+0000","last_clean":"2024-04-25T11:20:59.945079+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:20:59.945079+0000","last_undegraded":"2024-04-25T11:20:59.945079+0000","last_fullsized":"2024-04-25T11:20:59.945079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926127+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:20:59.926127+0000","last_peered":"2024-04-25T11:20:59.926127+0000","last_clean":"2024-04-25T11:20:59.926127+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:20:59.926127+0000","last_undegraded":"2024-04-25T11:20:59.926127+0000","last_fullsized":"2024-04-25T11:20:59.926127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.477014+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:21:00.477014+0000","last_peered":"2024-04-25T11:21:00.477014+0000","last_clean":"2024-04-25T11:21:00.477014+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:21:00.477014+0000","last_undegraded":"2024-04-25T11:21:00.477014+0000","last_fullsized":"2024-04-25T11:21:00.477014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474387+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:21:00.474387+0000","last_peered":"2024-04-25T11:21:00.474387+0000","last_clean":"2024-04-25T11:21:00.474387+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:21:00.474387+0000","last_undegraded":"2024-04-25T11:21:00.474387+0000","last_fullsized":"2024-04-25T11:21:00.474387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.933616+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:20:59.933616+0000","last_peered":"2024-04-25T11:20:59.933616+0000","last_clean":"2024-04-25T11:20:59.933616+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:20:59.933616+0000","last_undegraded":"2024-04-25T11:20:59.933616+0000","last_fullsized":"2024-04-25T11:20:59.933616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.935312+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:20:59.935312+0000","last_peered":"2024-04-25T11:20:59.935312+0000","last_clean":"2024-04-25T11:20:59.935312+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:20:59.935312+0000","last_undegraded":"2024-04-25T11:20:59.935312+0000","last_fullsized":"2024-04-25T11:20:59.935312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.932719+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:20:59.932719+0000","last_peered":"2024-04-25T11:20:59.932719+0000","last_clean":"2024-04-25T11:20:59.932719+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:20:59.932719+0000","last_undegraded":"2024-04-25T11:20:59.932719+0000","last_fullsized":"2024-04-25T11:20:59.932719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.068007+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:21:00.068007+0000","last_peered":"2024-04-25T11:21:00.068007+0000","last_clean":"2024-04-25T11:21:00.068007+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:21:00.068007+0000","last_undegraded":"2024-04-25T11:21:00.068007+0000","last_fullsized":"2024-04-25T11:21:00.068007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.925200+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:20:59.925200+0000","last_peered":"2024-04-25T11:20:59.925200+0000","last_clean":"2024-04-25T11:20:59.925200+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:20:59.925200+0000","last_undegraded":"2024-04-25T11:20:59.925200+0000","last_fullsized":"2024-04-25T11:20:59.925200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.924837+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:20:59.924837+0000","last_peered":"2024-04-25T11:20:59.924837+0000","last_clean":"2024-04-25T11:20:59.924837+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:20:59.924837+0000","last_undegraded":"2024-04-25T11:20:59.924837+0000","last_fullsized":"2024-04-25T11:20:59.924837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.111746+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:21:00.111746+0000","last_peered":"2024-04-25T11:21:00.111746+0000","last_clean":"2024-04-25T11:21:00.111746+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:21:00.111746+0000","last_undegraded":"2024-04-25T11:21:00.111746+0000","last_fullsized":"2024-04-25T11:21:00.111746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.926148+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:20:59.926148+0000","last_peered":"2024-04-25T11:20:59.926148+0000","last_clean":"2024-04-25T11:20:59.926148+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:20:59.926148+0000","last_undegraded":"2024-04-25T11:20:59.926148+0000","last_fullsized":"2024-04-25T11:20:59.926148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109471+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:21:00.109471+0000","last_peered":"2024-04-25T11:21:00.109471+0000","last_clean":"2024-04-25T11:21:00.109471+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:21:00.109471+0000","last_undegraded":"2024-04-25T11:21:00.109471+0000","last_fullsized":"2024-04-25T11:21:00.109471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474518+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:21:00.474518+0000","last_peered":"2024-04-25T11:21:00.474518+0000","last_clean":"2024-04-25T11:21:00.474518+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:21:00.474518+0000","last_undegraded":"2024-04-25T11:21:00.474518+0000","last_fullsized":"2024-04-25T11:21:00.474518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109512+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:21:00.109512+0000","last_peered":"2024-04-25T11:21:00.109512+0000","last_clean":"2024-04-25T11:21:00.109512+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:21:00.109512+0000","last_undegraded":"2024-04-25T11:21:00.109512+0000","last_fullsized":"2024-04-25T11:21:00.109512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.474969+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:21:00.474969+0000","last_peered":"2024-04-25T11:21:00.474969+0000","last_clean":"2024-04-25T11:21:00.474969+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:21:00.474969+0000","last_undegraded":"2024-04-25T11:21:00.474969+0000","last_fullsized":"2024-04-25T11:21:00.474969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.938661+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:20:59.938661+0000","last_peered":"2024-04-25T11:20:59.938661+0000","last_clean":"2024-04-25T11:20:59.938661+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:20:59.938661+0000","last_undegraded":"2024-04-25T11:20:59.938661+0000","last_fullsized":"2024-04-25T11:20:59.938661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:20:59.939739+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:20:59.939739+0000","last_peered":"2024-04-25T11:20:59.939739+0000","last_clean":"2024-04-25T11:20:59.939739+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:20:59.939739+0000","last_undegraded":"2024-04-25T11:20:59.939739+0000","last_fullsized":"2024-04-25T11:20:59.939739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T11:21:00.109891+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:21:00.109891+0000","last_peered":"2024-04-25T11:21:00.109891+0000","last_clean":"2024-04-25T11:21:00.109891+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:21:00.109891+0000","last_undegraded":"2024-04-25T11:21:00.109891+0000","last_fullsized":"2024-04-25T11:21:00.109891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12028,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":707270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":119875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31496,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340344,"statfs":{"total":96636764160,"available":96604512256,"internally_reserved":0,"allocated":327680,"data_stored":118734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":303104,"data_stored":116132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":119909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":116264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":115893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":901120,"data_stored":705603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:21:27.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:27.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 Processor -- start 2024-04-25T11:21:27.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 -- start start 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa5fc15c0d0 0x7fa5fc0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa5fc15ff60 0x7fa5fc0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5fc15c6f0 con 0x7fa5fc160370 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa5fc0c6310 con 0x7fa5fc15ff60 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa6008af640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa5fc15c990 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa5fc15ff60 0x7fa5fc0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5fad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa5fc160370 0x7fa5fc071590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48132/0 (socket says 172.21.15.136:48132) 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa5fc15ff60 0x7fa5fc0ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37112/0 (socket says 172.21.15.136:37112) 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5fad9d640 1 -- 172.21.15.136:0/4166589935 learned_addr learned my addr 172.21.15.136:0/4166589935 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5fa59c640 1 --2- 172.21.15.136:0/4166589935 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc15c0d0 0x7fa5fc0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 -- 172.21.15.136:0/4166589935 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc15c0d0 msgr2=0x7fa5fc0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/4166589935 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc15c0d0 0x7fa5fc0cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 -- 172.21.15.136:0/4166589935 >> v1:172.21.15.136:6789/0 conn(0x7fa5fc160370 legacy=0x7fa5fc071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.627+0000 7fa5f9d9b640 1 -- 172.21.15.136:0/4166589935 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc15cc30 con 0x7fa5fc15ff60 2024-04-25T11:21:27.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/4166589935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc0ce260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0734b0 tx=0x7fa5f001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6078424a6cb0b296 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa5f959a640 1 -- 172.21.15.136:0/4166589935 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00b5070 con 0x7fa5fc15ff60 2024-04-25T11:21:27.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa5f959a640 1 -- 172.21.15.136:0/4166589935 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f00a8970 con 0x7fa5fc15ff60 2024-04-25T11:21:27.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa5f959a640 1 -- 172.21.15.136:0/4166589935 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00a8d40 con 0x7fa5fc15ff60 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 -- 172.21.15.136:0/4166589935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15ff60 msgr2=0x7fa5fc0ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 --2- 172.21.15.136:0/4166589935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc0ce260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0734b0 tx=0x7fa5f001ec70 comp rx=0 tx=0).stop 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 -- 172.21.15.136:0/4166589935 shutdown_connections 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 --2- 172.21.15.136:0/4166589935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc0ce260 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 --2- 172.21.15.136:0/4166589935 >> v2:172.21.15.157:3300/0 conn(0x7fa5fc15c0d0 0x7fa5fc0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 -- 172.21.15.136:0/4166589935 >> 172.21.15.136:0/4166589935 conn(0x7fa5fc06f9d0 msgr2=0x7fa5fc06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 -- 172.21.15.136:0/4166589935 shutdown_connections 2024-04-25T11:21:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.631+0000 7fa6008af640 1 -- 172.21.15.136:0/4166589935 wait complete. 2024-04-25T11:21:27.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 Processor -- start 2024-04-25T11:21:27.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- start start 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc1f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 0x7fa5fc1f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc0cbc60 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc15fa40 con 0x7fa5fc15ff60 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa5fc15cf10 con 0x7fa5fc160370 2024-04-25T11:21:27.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37122/0 (socket says 172.21.15.136:37122) 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 -- 172.21.15.136:0/478875302 learned_addr learned my addr 172.21.15.136:0/478875302 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc1f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fad9d640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 0x7fa5fc1f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 -- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 msgr2=0x7fa5fc1f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 0x7fa5fc1f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 -- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 msgr2=0x7fa5fc1f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc1f6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 -- 172.21.15.136:0/478875302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc15d910 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fad9d640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 0x7fa5fc1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc1f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5fa59c640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4007e70 tx=0x7fa5f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4014620 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f4019070 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4014a00 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc15ee90 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc15de70 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.635+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5fc06a350 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.639+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa5f4014ba0 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.639+0000 7fa5eaffd640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.639+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa5f4090ae0 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.639+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.639+0000 7fa5f9d9b640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 0x7fa5dc071940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2000 tx=0x7fa5f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.643+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa5fc15d910 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.795+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5fc0c7bb0 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa5eaffd640 1 -- 172.21.15.136:0/478875302 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7fa5f405d5b0 con 0x7fa5fc15c0d0 2024-04-25T11:21:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:27.798 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 msgr2=0x7fa5dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 0x7fa5dc071940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2000 tx=0x7fa5f0008040 comp rx=0 tx=0).stop 2024-04-25T11:21:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 msgr2=0x7fa5fc153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4007e70 tx=0x7fa5f400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 shutdown_connections 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.799+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa5fc160370 0x7fa5fc1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.803+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5fc15ff60 0x7fa5fc1f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.803+0000 7fa6008af640 1 --2- 172.21.15.136:0/478875302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc15c0d0 0x7fa5fc153820 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.803+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 >> 172.21.15.136:0/478875302 conn(0x7fa5fc06f9d0 msgr2=0x7fa5fc0ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.803+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 shutdown_connections 2024-04-25T11:21:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.803+0000 7fa6008af640 1 -- 172.21.15.136:0/478875302 wait complete. 2024-04-25T11:21:27.818 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:21:27.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:21:27.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.955+0000 7f2ec10c8640 1 Processor -- start 2024-04-25T11:21:27.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ec10c8640 1 -- start start 2024-04-25T11:21:27.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ec10c8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 0x7f2ebc0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ec10c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc159900 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ec10c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ebc15c3e0 con 0x7f2ebc1615c0 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ec10c8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ebc1596d0 con 0x7f2ebc0c4750 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ebad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2ebc0c7c50 0x7f2ebc0c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48140/0 (socket says 172.21.15.136:48140) 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ebad9d640 1 -- 172.21.15.136:0/2305049758 learned_addr learned my addr 172.21.15.136:0/2305049758 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 0x7f2ebc0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3326114213 0 0) 0x7f2ebc159900 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea8003530 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1499559605 0 0) 0x7f2ea8003530 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 msgr2=0x7f2ebc0c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 0x7f2ebc0c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 >> v1:172.21.15.157:6789/0 conn(0x7f2ebc1615c0 legacy=0x7f2ebc1639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc15c6c0 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f2eac001680 con 0x7f2ebc0c7c50 2024-04-25T11:21:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 >> v1:172.21.15.136:6789/0 conn(0x7f2ebc0c7c50 legacy=0x7f2ebc0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 0x7f2ea80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ea8003f70 0x7f2ea8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ea8004d50 0x7f2ea8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea800bbf0 con 0x7f2ea8003f70 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea800bd70 con 0x7f2ea8004d50 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea800bf70 con 0x7f2ebc0c4750 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ea8003f70 0x7f2ea8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2ebb59e640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ea8004d50 0x7f2ea8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 -- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 msgr2=0x7f2ea80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc0c4750 0x7f2ea80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 -- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ea8004d50 msgr2=0x7f2ea8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ea8004d50 0x7f2ea8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.959+0000 7f2eb3fff640 1 -- 172.21.15.136:0/2305049758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea8003530 con 0x7f2ea8003f70 2024-04-25T11:21:27.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ea8003f70 0x7f2ea8004810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc15d360 tx=0x7f2ea401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924e1b6ac29e0f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40b2070 con 0x7f2ea8003f70 2024-04-25T11:21:27.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea40a5640 con 0x7f2ea8003f70 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2eba59c640 1 -- 172.21.15.136:0/2305049758 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40096b0 con 0x7f2ea8003f70 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- 172.21.15.136:0/2305049758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ea8003f70 msgr2=0x7f2ea8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/2305049758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ea8003f70 0x7f2ea8004810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc15d360 tx=0x7f2ea401ca50 comp rx=0 tx=0).stop 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ebb59e640 1 -- 172.21.15.136:0/2305049758 reap_dead start 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- 172.21.15.136:0/2305049758 shutdown_connections 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- 172.21.15.136:0/2305049758 >> 172.21.15.136:0/2305049758 conn(0x7f2ebc0c9d50 msgr2=0x7f2ebc0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- 172.21.15.136:0/2305049758 shutdown_connections 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- 172.21.15.136:0/2305049758 wait complete. 2024-04-25T11:21:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 Processor -- start 2024-04-25T11:21:27.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.963+0000 7f2ec10c8640 1 -- start start 2024-04-25T11:21:27.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc152100 0x7f2ebc14ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ebc148710 0x7f2ebc14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc15cc80 con 0x7f2ebc148b20 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ebc159bc0 con 0x7f2ebc148710 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ebc0c5270 con 0x7f2ebc152100 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37138/0 (socket says 172.21.15.136:37138) 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 -- 172.21.15.136:0/3249400788 learned_addr learned my addr 172.21.15.136:0/3249400788 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2eb3fff640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ebc148710 0x7f2ebc14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebad9d640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc152100 0x7f2ebc14ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 -- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc152100 msgr2=0x7f2ebc14ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc152100 0x7f2ebc14ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 -- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ebc148710 msgr2=0x7f2ebc14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ebc148710 0x7f2ebc14f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 -- 172.21.15.136:0/3249400788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ebc14d640 con 0x7f2ebc148b20 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ebb59e640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc072080 tx=0x7f2eb400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400ee90 con 0x7f2ebc148b20 2024-04-25T11:21:27.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb4006b60 con 0x7f2ebc148b20 2024-04-25T11:21:27.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4006d30 con 0x7f2ebc148b20 2024-04-25T11:21:27.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ebc14d920 con 0x7f2ebc148b20 2024-04-25T11:21:27.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ebc14dee0 con 0x7f2ebc148b20 2024-04-25T11:21:27.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ebc06a350 con 0x7f2ebc148b20 2024-04-25T11:21:27.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.967+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2eb401a020 con 0x7f2ebc148b20 2024-04-25T11:21:27.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.971+0000 7f2eb37fe640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 0x7f2e9c071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:27.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.971+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2eb408ebe0 con 0x7f2ebc148b20 2024-04-25T11:21:27.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.971+0000 7f2ebad9d640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 0x7f2e9c071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:27.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.971+0000 7f2ebad9d640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 0x7f2e9c071ae0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001870 tx=0x7f2eac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:27.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:27.975+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2eb405b760 con 0x7f2ebc148b20 2024-04-25T11:21:28.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.139+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ebc0699a0 con 0x7f2ebc148b20 2024-04-25T11:21:28.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2eb37fe640 1 -- 172.21.15.136:0/3249400788 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f2eb405b060 con 0x7f2ebc148b20 2024-04-25T11:21:28.140 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:21:28.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 msgr2=0x7f2e9c071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 0x7f2e9c071ae0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001870 tx=0x7f2eac009040 comp rx=0 tx=0).stop 2024-04-25T11:21:28.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 msgr2=0x7f2ebc14fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc072080 tx=0x7f2eb400eb50 comp rx=0 tx=0).stop 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 shutdown_connections 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ebc148b20 0x7f2ebc14fe70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ebc148710 0x7f2ebc14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e9c06f5d0 0x7f2e9c071ae0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 --2- 172.21.15.136:0/3249400788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ebc152100 0x7f2ebc14ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 >> 172.21.15.136:0/3249400788 conn(0x7f2ebc0c9d50 msgr2=0x7f2ebc06fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 shutdown_connections 2024-04-25T11:21:28.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.143+0000 7f2ec10c8640 1 -- 172.21.15.136:0/3249400788 wait complete. 2024-04-25T11:21:28.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:21:28.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.299+0000 7f9a6147a640 1 Processor -- start 2024-04-25T11:21:28.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 -- start start 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c1639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a5c159900 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9a5c1596d0 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a6147a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5c159bc0 con 0x7f9a5c1615c0 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37140/0 (socket says 172.21.15.136:37140) 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 -- 172.21.15.136:0/1620778021 learned_addr learned my addr 172.21.15.136:0/1620778021 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5b7fe640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c1639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 -- 172.21.15.136:0/1620778021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 msgr2=0x7f9a5c1639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c1639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 -- 172.21.15.136:0/1620778021 >> v1:172.21.15.157:6789/0 conn(0x7f9a5c0c8e40 legacy=0x7f9a5c0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:28.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5affd640 1 -- 172.21.15.136:0/1620778021 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c15c3e0 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.303+0000 7f9a5b7fe640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c1639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:28.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a5affd640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009870 tx=0x7f9a4c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb0c27ebea2a893 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a5a7fc640 1 -- 172.21.15.136:0/1620778021 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0b5070 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a5a7fc640 1 -- 172.21.15.136:0/1620778021 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c0a89d0 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a5a7fc640 1 -- 172.21.15.136:0/1620778021 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0a8da0 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- 172.21.15.136:0/1620778021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c7c50 msgr2=0x7f9a5c0c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009870 tx=0x7f9a4c01ed10 comp rx=0 tx=0).stop 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- 172.21.15.136:0/1620778021 shutdown_connections 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c1639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 --2- 172.21.15.136:0/1620778021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c0c8900 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- 172.21.15.136:0/1620778021 >> 172.21.15.136:0/1620778021 conn(0x7f9a5c0c9d50 msgr2=0x7f9a5c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- 172.21.15.136:0/1620778021 shutdown_connections 2024-04-25T11:21:28.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- 172.21.15.136:0/1620778021 wait complete. 2024-04-25T11:21:28.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 Processor -- start 2024-04-25T11:21:28.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.307+0000 7f9a6147a640 1 -- start start 2024-04-25T11:21:28.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c1f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c15cc80 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c15c9a0 con 0x7f9a5c0c7c50 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5c0c5270 con 0x7f9a5c1615c0 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37144/0 (socket says 172.21.15.136:37144) 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 -- 172.21.15.136:0/495427483 learned_addr learned my addr 172.21.15.136:0/495427483 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a5b7fe640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a5affd640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c1f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 -- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 msgr2=0x7f9a5c145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c145a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 -- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 msgr2=0x7f9a5c1f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c1f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 -- 172.21.15.136:0/495427483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c1f4290 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a5affd640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a527fc640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400b3f0 tx=0x7f9a4400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44015020 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a44011830 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c1f0850 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44011c00 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c1f0e10 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c06a350 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a44018020 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 0x7f9a3c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9a4408f340 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.311+0000 7f9a5affd640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 0x7f9a3c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.315+0000 7f9a5affd640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 0x7f9a3c071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0b2000 tx=0x7f9a4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.315+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a4401f030 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.463+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a5c0c63e0 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.463+0000 7f9a53fff640 1 -- 172.21.15.136:0/495427483 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f9a4405be30 con 0x7f9a5c0c8e40 2024-04-25T11:21:28.464 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 msgr2=0x7f9a3c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 0x7f9a3c071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0b2000 tx=0x7f9a4c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 msgr2=0x7f9a5c0c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400b3f0 tx=0x7f9a4400b8b0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 shutdown_connections 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a3c06f400 0x7f9a3c071910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a5c1615c0 0x7f9a5c145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a5c0c8e40 0x7f9a5c0c6e50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 --2- 172.21.15.136:0/495427483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a5c0c7c50 0x7f9a5c1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 >> 172.21.15.136:0/495427483 conn(0x7f9a5c0c9d50 msgr2=0x7f9a5c0c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 shutdown_connections 2024-04-25T11:21:28.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.467+0000 7f9a6147a640 1 -- 172.21.15.136:0/495427483 wait complete. 2024-04-25T11:21:28.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:21:28.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 Processor -- start 2024-04-25T11:21:28.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 -- start start 2024-04-25T11:21:28.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f356c072dd0 0x7f356c0c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f356c15d590 con 0x7f356c0c8190 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f356c15d340 con 0x7f356c072dd0 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3572511640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f356c15e1a0 con 0x7f356c0729c0 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f356c072dd0 0x7f356c0c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f356c072dd0 0x7f356c0c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37154/0 (socket says 172.21.15.136:37154) 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 -- 172.21.15.136:0/3352204519 learned_addr learned my addr 172.21.15.136:0/3352204519 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f356bfff640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 -- 172.21.15.136:0/3352204519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 msgr2=0x7f356c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 -- 172.21.15.136:0/3352204519 >> v1:172.21.15.157:6789/0 conn(0x7f356c0c8190 legacy=0x7f356c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f3563fff640 1 -- 172.21.15.136:0/3352204519 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c15d870 con 0x7f356c072dd0 2024-04-25T11:21:28.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.635+0000 7f356bfff640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:28.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3563fff640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c0c7c50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3554009870 tx=0x7f355401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40359bcd7e3a38b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f356b7fe640 1 -- 172.21.15.136:0/3352204519 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540ab650 con 0x7f356c072dd0 2024-04-25T11:21:28.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f356b7fe640 1 -- 172.21.15.136:0/3352204519 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35540b0070 con 0x7f356c072dd0 2024-04-25T11:21:28.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f356b7fe640 1 -- 172.21.15.136:0/3352204519 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540aba10 con 0x7f356c072dd0 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 -- 172.21.15.136:0/3352204519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 msgr2=0x7f356c0c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c0c7c50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3554009870 tx=0x7f355401c560 comp rx=0 tx=0).stop 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 -- 172.21.15.136:0/3352204519 shutdown_connections 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c0c7c50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 --2- 172.21.15.136:0/3352204519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 -- 172.21.15.136:0/3352204519 >> 172.21.15.136:0/3352204519 conn(0x7f356c0c9d50 msgr2=0x7f356c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 -- 172.21.15.136:0/3352204519 shutdown_connections 2024-04-25T11:21:28.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.639+0000 7f3572511640 1 -- 172.21.15.136:0/3352204519 wait complete. 2024-04-25T11:21:28.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 Processor -- start 2024-04-25T11:21:28.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- start start 2024-04-25T11:21:28.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 0x7f356c13aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f356c143370 con 0x7f356c072dd0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f356c1431f0 con 0x7f356c0c8190 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f356c166ef0 con 0x7f356c0729c0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37160/0 (socket says 172.21.15.136:37160) 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 -- 172.21.15.136:0/185946671 learned_addr learned my addr 172.21.15.136:0/185946671 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f356bfff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3570aae640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 0x7f356c13aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 -- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 msgr2=0x7f356c13ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c13ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 -- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 msgr2=0x7f356c13aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 0x7f356c13aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 -- 172.21.15.136:0/185946671 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c15ba10 con 0x7f356c072dd0 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f356bfff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3570aae640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 0x7f356c13aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3563fff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35540ad040 tx=0x7f355401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540be5f0 con 0x7f356c072dd0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35540be7b0 con 0x7f356c072dd0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356c13cbd0 con 0x7f356c072dd0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35540beba0 con 0x7f356c072dd0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356c13d170 con 0x7f356c072dd0 2024-04-25T11:21:28.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.643+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356c06a350 con 0x7f356c072dd0 2024-04-25T11:21:28.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.647+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f35540c73f0 con 0x7f356c072dd0 2024-04-25T11:21:28.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.647+0000 7f35697fa640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 0x7f354c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.647+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f355412a3c0 con 0x7f356c072dd0 2024-04-25T11:21:28.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.647+0000 7f356bfff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 0x7f354c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.647+0000 7f356bfff640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 0x7f354c071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097a0 tx=0x7f355c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.651+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3554016150 con 0x7f356c072dd0 2024-04-25T11:21:28.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.811+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f356c0cc8c0 con 0x7f356c072dd0 2024-04-25T11:21:28.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.811+0000 7f35697fa640 1 -- 172.21.15.136:0/185946671 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f35540f70b0 con 0x7f356c072dd0 2024-04-25T11:21:28.810 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:21:28.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 msgr2=0x7f354c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 0x7f354c071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097a0 tx=0x7f355c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 msgr2=0x7f356c15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f35540ad040 tx=0x7f355401ada0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 shutdown_connections 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f356c0c8190 0x7f356c13aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f356c072dd0 0x7f356c15b4d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f354c06f430 0x7f354c071940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 --2- 172.21.15.136:0/185946671 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f356c0729c0 0x7f356c13ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 >> 172.21.15.136:0/185946671 conn(0x7f356c0c9d50 msgr2=0x7f356c0cb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 shutdown_connections 2024-04-25T11:21:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.815+0000 7f3572511640 1 -- 172.21.15.136:0/185946671 wait complete. 2024-04-25T11:21:28.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:21:28.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 Processor -- start 2024-04-25T11:21:28.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 -- start start 2024-04-25T11:21:28.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc35c0c96e0 0x7fc35c1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 0x7fc35c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c0c9af0 con 0x7fc35c15a3d0 2024-04-25T11:21:28.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc35c06cc10 con 0x7fc35c0c96e0 2024-04-25T11:21:28.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc36109a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0cb220 con 0x7fc35c0c79e0 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc353fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 0x7fc35c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc35c0c96e0 0x7fc35c1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc353fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 0x7fc35c0c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56316/0 (socket says 172.21.15.136:56316) 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc353fff640 1 -- 172.21.15.136:0/500741207 learned_addr learned my addr 172.21.15.136:0/500741207 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc35c0c96e0 0x7fc35c1540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:57242/0 (socket says 172.21.15.136:57242) 2024-04-25T11:21:28.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 -- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 msgr2=0x7fc35c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:28.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 --2- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 0x7fc35c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 -- 172.21.15.136:0/500741207 >> v1:172.21.15.157:6789/0 conn(0x7fc35c15a3d0 legacy=0x7fc35c15a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.987+0000 7fc35ad9d640 1 -- 172.21.15.136:0/500741207 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c0cb500 con 0x7fc35c0c96e0 2024-04-25T11:21:28.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc35ad9d640 1 --2- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c96e0 0x7fc35c1540d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0097c0 tx=0x7fc34c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8088552e6b0178ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc35a59c640 1 -- 172.21.15.136:0/500741207 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0b5070 con 0x7fc35c0c96e0 2024-04-25T11:21:28.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc35a59c640 1 -- 172.21.15.136:0/500741207 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c0a8990 con 0x7fc35c0c96e0 2024-04-25T11:21:28.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc35a59c640 1 -- 172.21.15.136:0/500741207 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c0a8d60 con 0x7fc35c0c96e0 2024-04-25T11:21:28.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 -- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c96e0 msgr2=0x7fc35c1540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 --2- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c96e0 0x7fc35c1540d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0097c0 tx=0x7fc34c01ec90 comp rx=0 tx=0).stop 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 -- 172.21.15.136:0/500741207 shutdown_connections 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 --2- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0c79e0 0x7fc35c0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 --2- 172.21.15.136:0/500741207 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c96e0 0x7fc35c1540d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 -- 172.21.15.136:0/500741207 >> 172.21.15.136:0/500741207 conn(0x7fc35c06f9d0 msgr2=0x7fc35c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 -- 172.21.15.136:0/500741207 shutdown_connections 2024-04-25T11:21:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.991+0000 7fc36109a640 1 -- 172.21.15.136:0/500741207 wait complete. 2024-04-25T11:21:28.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 Processor -- start 2024-04-25T11:21:28.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 -- start start 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c79e0 0x7fc35c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c15a3d0 0x7fc35c0c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0cf0f0 con 0x7fc35c0c96e0 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c154610 con 0x7fc35c0c79e0 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc36109a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0ced40 con 0x7fc35c15a3d0 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37162/0 (socket says 172.21.15.136:37162) 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 -- 172.21.15.136:0/1160401778 learned_addr learned my addr 172.21.15.136:0/1160401778 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc35ad9d640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c79e0 0x7fc35c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc35b59e640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c15a3d0 0x7fc35c0c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 -- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c15a3d0 msgr2=0x7fc35c0c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c15a3d0 0x7fc35c0c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 -- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c79e0 msgr2=0x7fc35c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c79e0 0x7fc35c1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 -- 172.21.15.136:0/1160401778 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c158810 con 0x7fc35c0c96e0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.995+0000 7fc353fff640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc34400e820 tx=0x7fc34400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc344017070 con 0x7fc35c0c96e0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34400c940 con 0x7fc35c0c96e0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c1f5f40 con 0x7fc35c0c96e0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34400cd30 con 0x7fc35c0c96e0 2024-04-25T11:21:28.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c1f6500 con 0x7fc35c0c96e0 2024-04-25T11:21:28.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35c1f5cb0 con 0x7fc35c0c96e0 2024-04-25T11:21:28.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc34401a070 con 0x7fc35c0c96e0 2024-04-25T11:21:28.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 0x7fc33c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:28.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc34408f7c0 con 0x7fc35c0c96e0 2024-04-25T11:21:28.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc35ad9d640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 0x7fc33c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:28.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:28.999+0000 7fc35ad9d640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 0x7fc33c071910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0b2000 tx=0x7fc34c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.003+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc34405bd80 con 0x7fc35c0c96e0 2024-04-25T11:21:29.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc35c06a350 con 0x7fc35c0c96e0 2024-04-25T11:21:29.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc3537fe640 1 -- 172.21.15.136:0/1160401778 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc344060080 con 0x7fc35c0c96e0 2024-04-25T11:21:29.165 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 msgr2=0x7fc33c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 0x7fc33c071910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc34c0b2000 tx=0x7fc34c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 msgr2=0x7fc35c15da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc34400e820 tx=0x7fc34400ece0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 shutdown_connections 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc33c06f400 0x7fc33c071910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.167+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c15a3d0 0x7fc35c0c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.171+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc35c0c96e0 0x7fc35c15da30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.171+0000 7fc36109a640 1 --2- 172.21.15.136:0/1160401778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0c79e0 0x7fc35c1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.171+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 >> 172.21.15.136:0/1160401778 conn(0x7fc35c06f9d0 msgr2=0x7fc35c152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:29.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.171+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 shutdown_connections 2024-04-25T11:21:29.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.171+0000 7fc36109a640 1 -- 172.21.15.136:0/1160401778 wait complete. 2024-04-25T11:21:29.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:21:29.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.339+0000 7fa63d8fb640 1 Processor -- start 2024-04-25T11:21:29.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.339+0000 7fa63d8fb640 1 -- start start 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa63d8fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 0x7fa638159f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa63d8fb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6380689a0 0x7fa638153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa63d8fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6380b3720 con 0x7fa638068590 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa63d8fb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa638073b30 con 0x7fa6380689a0 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa63d8fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6380b3a00 con 0x7fa638074970 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6380689a0 0x7fa638153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6380689a0 0x7fa638153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37172/0 (socket says 172.21.15.136:37172) 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 -- 172.21.15.136:0/2095012480 learned_addr learned my addr 172.21.15.136:0/2095012480 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa636ffd640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 0x7fa638159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 -- 172.21.15.136:0/2095012480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 msgr2=0x7fa638159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:29.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 0x7fa638159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 -- 172.21.15.136:0/2095012480 >> v1:172.21.15.157:6789/0 conn(0x7fa638068590 legacy=0x7fa638073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa6377fe640 1 -- 172.21.15.136:0/2095012480 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6380b3ce0 con 0x7fa6380689a0 2024-04-25T11:21:29.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.343+0000 7fa636ffd640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 0x7fa638159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:21:29.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa6377fe640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa638153ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa638073cb0 tx=0x7fa63001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2777605cfed21daa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa635ffb640 1 -- 172.21.15.136:0/2095012480 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6300a56e0 con 0x7fa6380689a0 2024-04-25T11:21:29.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa635ffb640 1 -- 172.21.15.136:0/2095012480 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6300a5880 con 0x7fa6380689a0 2024-04-25T11:21:29.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa635ffb640 1 -- 172.21.15.136:0/2095012480 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6300a5c50 con 0x7fa6380689a0 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- 172.21.15.136:0/2095012480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 msgr2=0x7fa638153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa638153ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa638073cb0 tx=0x7fa63001ca10 comp rx=0 tx=0).stop 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- 172.21.15.136:0/2095012480 shutdown_connections 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa638153ac0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/2095012480 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638074970 0x7fa638159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- 172.21.15.136:0/2095012480 >> 172.21.15.136:0/2095012480 conn(0x7fa6380ca640 msgr2=0x7fa6380caa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- 172.21.15.136:0/2095012480 shutdown_connections 2024-04-25T11:21:29.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- 172.21.15.136:0/2095012480 wait complete. 2024-04-25T11:21:29.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 Processor -- start 2024-04-25T11:21:29.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.347+0000 7fa63d8fb640 1 -- start start 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 0x7fa638144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa638074970 0x7fa638142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa63815a490 con 0x7fa6380689a0 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380cb440 con 0x7fa638074970 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6381670f0 con 0x7fa638068590 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37178/0 (socket says 172.21.15.136:37178) 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 -- 172.21.15.136:0/804320881 learned_addr learned my addr 172.21.15.136:0/804320881 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa636ffd640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 0x7fa638144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 -- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 msgr2=0x7fa638144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6377fe640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa638074970 0x7fa638142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 0x7fa638144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 -- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa638074970 msgr2=0x7fa638142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa638074970 0x7fa638142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 -- 172.21.15.136:0/804320881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63815a180 con 0x7fa6380689a0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa636ffd640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 0x7fa638144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa6367fc640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00c340 tx=0x7fa62c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c0052d0 con 0x7fa6380689a0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62c005470 con 0x7fa6380689a0 2024-04-25T11:21:29.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c007260 con 0x7fa6380689a0 2024-04-25T11:21:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6381f3730 con 0x7fa6380689a0 2024-04-25T11:21:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6381f3cf0 con 0x7fa6380689a0 2024-04-25T11:21:29.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa638159ed0 con 0x7fa6380689a0 2024-04-25T11:21:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa62c007400 con 0x7fa6380689a0 2024-04-25T11:21:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 0x7fa624071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa62c08eb40 con 0x7fa6380689a0 2024-04-25T11:21:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.351+0000 7fa636ffd640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 0x7fa624071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.355+0000 7fa636ffd640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 0x7fa624071910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa638071e00 tx=0x7fa628008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.359+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa62c05b630 con 0x7fa6380689a0 2024-04-25T11:21:29.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.555+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa63806a350 con 0x7fa6380689a0 2024-04-25T11:21:29.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.555+0000 7fa61ffff640 1 -- 172.21.15.136:0/804320881 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7fa62c05af30 con 0x7fa6380689a0 2024-04-25T11:21:29.555 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:29.555 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:21:29.555 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 msgr2=0x7fa624071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 0x7fa624071910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa638071e00 tx=0x7fa628008040 comp rx=0 tx=0).stop 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 msgr2=0x7fa6380c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00c340 tx=0x7fa62c00c800 comp rx=0 tx=0).stop 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 shutdown_connections 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa638074970 0x7fa638142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6380689a0 0x7fa6380c7160 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa62406f400 0x7fa624071910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 --2- 172.21.15.136:0/804320881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa638068590 0x7fa638144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 >> 172.21.15.136:0/804320881 conn(0x7fa6380ca640 msgr2=0x7fa6380c9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:29.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 shutdown_connections 2024-04-25T11:21:29.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.559+0000 7fa63d8fb640 1 -- 172.21.15.136:0/804320881 wait complete. 2024-04-25T11:21:29.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:21:29.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 Processor -- start 2024-04-25T11:21:29.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 -- start start 2024-04-25T11:21:29.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f78940c79e0 0x7f78940c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78940c9af0 con 0x7f78940c96e0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f789406cc10 con 0x7f78940c79e0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789c53f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78940cb220 con 0x7f789415a3d0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f78940c79e0 0x7f78940c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f78940c79e0 0x7f78940c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37188/0 (socket says 172.21.15.136:37188) 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 -- 172.21.15.136:0/2464176579 learned_addr learned my addr 172.21.15.136:0/2464176579 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789aadc640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 -- 172.21.15.136:0/2464176579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 msgr2=0x7f789415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789415a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 -- 172.21.15.136:0/2464176579 >> v1:172.21.15.157:6789/0 conn(0x7f78940c96e0 legacy=0x7f78941540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f7899ada640 1 -- 172.21.15.136:0/2464176579 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940cb500 con 0x7f78940c79e0 2024-04-25T11:21:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.723+0000 7f789aadc640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:29.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f7899ada640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c79e0 0x7f78940c7df0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f78940c8990 tx=0x7f788401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be66db3fabf304d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f78992d9640 1 -- 172.21.15.136:0/2464176579 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78840b0070 con 0x7f78940c79e0 2024-04-25T11:21:29.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f78992d9640 1 -- 172.21.15.136:0/2464176579 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78840ab600 con 0x7f78940c79e0 2024-04-25T11:21:29.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f78992d9640 1 -- 172.21.15.136:0/2464176579 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78840ab9d0 con 0x7f78940c79e0 2024-04-25T11:21:29.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 -- 172.21.15.136:0/2464176579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c79e0 msgr2=0x7f78940c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c79e0 0x7f78940c7df0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f78940c8990 tx=0x7f788401c4e0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 -- 172.21.15.136:0/2464176579 shutdown_connections 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 --2- 172.21.15.136:0/2464176579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c79e0 0x7f78940c7df0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 -- 172.21.15.136:0/2464176579 >> 172.21.15.136:0/2464176579 conn(0x7f789406f9d0 msgr2=0x7f789406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 -- 172.21.15.136:0/2464176579 shutdown_connections 2024-04-25T11:21:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.727+0000 7f789c53f640 1 -- 172.21.15.136:0/2464176579 wait complete. 2024-04-25T11:21:29.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 Processor -- start 2024-04-25T11:21:29.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- start start 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 0x7f7894159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789413eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f78940ced40 con 0x7f78940c96e0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f789406d440 con 0x7f78940c79e0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7894157dd0 con 0x7f789415a3d0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37194/0 (socket says 172.21.15.136:37194) 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 -- 172.21.15.136:0/1356162758 learned_addr learned my addr 172.21.15.136:0/1356162758 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789aadc640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789413eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789a2db640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 0x7f7894159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 -- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 msgr2=0x7f789413eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789413eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 -- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 msgr2=0x7f7894159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 0x7f7894159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 -- 172.21.15.136:0/1356162758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7894159cb0 con 0x7f78940c96e0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789aadc640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789413eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789a2db640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 0x7f7894159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f7899ada640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f788401cd80 tx=0x7f788401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78840b0040 con 0x7f78940c96e0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78840bc070 con 0x7f78940c96e0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78840b6880 con 0x7f78940c96e0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78940cd4d0 con 0x7f78940c96e0 2024-04-25T11:21:29.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.731+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78940cda90 con 0x7f78940c96e0 2024-04-25T11:21:29.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78940cd220 con 0x7f78940c96e0 2024-04-25T11:21:29.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f78840b6a20 con 0x7f78940c96e0 2024-04-25T11:21:29.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f788affd640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 0x7f7878071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:21:29.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7884129cf0 con 0x7f78940c96e0 2024-04-25T11:21:29.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f789a2db640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 0x7f7878071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:21:29.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.735+0000 7f789a2db640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 0x7f7878071940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f788c00a850 tx=0x7f788c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:21:29.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.739+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f78840160d0 con 0x7f78940c96e0 2024-04-25T11:21:29.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.891+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f789406a350 con 0x7f78940c96e0 2024-04-25T11:21:29.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f788affd640 1 -- 172.21.15.136:0/1356162758 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18939 (secure 0 0 0) 0x7f78840f68e0 con 0x7f78940c96e0 2024-04-25T11:21:29.893 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:21:29.893 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:20:59.917869+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 msgr2=0x7f7878071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 0x7f7878071940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f788c00a850 tx=0x7f788c008040 comp rx=0 tx=0).stop 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 msgr2=0x7f78941445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f788401cd80 tx=0x7f788401ad20 comp rx=0 tx=0).stop 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 shutdown_connections 2024-04-25T11:21:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f787806f430 0x7f7878071940 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f789415a3d0 0x7f789413eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78940c96e0 0x7f78941445b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 --2- 172.21.15.136:0/1356162758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78940c79e0 0x7f7894159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 >> 172.21.15.136:0/1356162758 conn(0x7f789406f9d0 msgr2=0x7f7894152f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 shutdown_connections 2024-04-25T11:21:29.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:21:29.895+0000 7f789c53f640 1 -- 172.21.15.136:0/1356162758 wait complete. 2024-04-25T11:21:29.913 DEBUG:tasks.mds_thrash:Yielding 2024-04-25T11:21:29.914 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T11:21:29.927 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:21:29.928 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T11:21:29.928 INFO:tasks.workunit:timeout=3h 2024-04-25T11:21:29.928 INFO:tasks.workunit:cleanup=True 2024-04-25T11:21:29.928 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:29.929 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-04-25T11:21:29.929 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 100 secs before thrashing 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-25 11:20:25.034098343 +0000 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-25 11:20:37.534331021 +0000 2024-04-25T11:21:29.935 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-25T11:21:29.935 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T11:21:29.936 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T11:21:30.000 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T11:21:30.050 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T11:21:49.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:21:49.640 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:22:19.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:22:19.687 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:22:49.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:22:49.746 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:22:51.383 INFO:tasks.workunit.client.0.smithi136.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr: git switch -c 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.384 INFO:tasks.workunit.client.0.smithi136.stderr:Or undo this operation with: 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr: git switch - 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr: 2024-04-25T11:22:51.385 INFO:tasks.workunit.client.0.smithi136.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T11:22:51.391 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T11:22:51.397 INFO:tasks.workunit.client.0.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T11:22:51.401 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T11:22:51.401 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T11:22:52.204 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T11:22:52.293 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T11:22:52.345 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T11:22:52.347 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T11:22:52.347 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T11:22:52.393 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T11:22:52.397 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:22:52.397 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T11:22:52.443 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-25T11:22:52.444 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-25T11:22:52.444 DEBUG:teuthology.orchestra.run.smithi136:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T11:22:52.506 INFO:tasks.workunit.client.0.smithi136.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T11:22:52.507 INFO:tasks.workunit.client.0.smithi136.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-25T11:22:52.507 INFO:tasks.workunit.client.0.smithi136.stderr:+ trap -- retry EXIT 2024-04-25T11:22:52.507 INFO:tasks.workunit.client.0.smithi136.stderr:+ rm -rf ffsb 2024-04-25T11:22:52.508 INFO:tasks.workunit.client.0.smithi136.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-25T11:22:52.514 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into 'ffsb'... 2024-04-25T11:22:56.551 INFO:tasks.workunit.client.0.smithi136.stderr:+ trap - EXIT 2024-04-25T11:22:56.551 INFO:tasks.workunit.client.0.smithi136.stderr:+ cd ffsb 2024-04-25T11:22:56.552 INFO:tasks.workunit.client.0.smithi136.stderr:+ ./configure 2024-04-25T11:22:56.762 INFO:tasks.workunit.client.0.smithi136.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T11:22:57.781 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether build environment is sane... yes 2024-04-25T11:22:57.804 INFO:tasks.workunit.client.0.smithi136.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-25T11:22:57.804 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gawk... gawk 2024-04-25T11:22:57.844 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T11:22:57.914 INFO:tasks.workunit.client.0.smithi136.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T11:22:57.914 INFO:tasks.workunit.client.0.smithi136.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T11:22:57.956 INFO:tasks.workunit.client.0.smithi136.stdout:checking for style of include used by make... GNU 2024-04-25T11:22:57.965 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc... gcc 2024-04-25T11:22:58.085 INFO:tasks.workunit.client.0.smithi136.stdout:checking for C compiler default output file name... a.out 2024-04-25T11:22:58.089 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether the C compiler works... yes 2024-04-25T11:22:58.095 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether we are cross compiling... no 2024-04-25T11:22:58.164 INFO:tasks.workunit.client.0.smithi136.stdout:checking for suffix of executables... 2024-04-25T11:22:58.237 INFO:tasks.workunit.client.0.smithi136.stdout:checking for suffix of object files... o 2024-04-25T11:22:58.296 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether we are using the GNU C compiler... yes 2024-04-25T11:22:58.356 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether gcc accepts -g... yes 2024-04-25T11:22:58.493 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-25T11:22:58.846 INFO:tasks.workunit.client.0.smithi136.stdout:checking dependency style of gcc... gcc3 2024-04-25T11:22:58.981 INFO:tasks.workunit.client.0.smithi136.stdout:checking how to run the C preprocessor... gcc -E 2024-04-25T11:22:59.143 INFO:tasks.workunit.client.0.smithi136.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T11:22:59.145 INFO:tasks.workunit.client.0.smithi136.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T11:22:59.166 INFO:tasks.workunit.client.0.smithi136.stdout:checking for AIX... no 2024-04-25T11:22:59.188 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc... (cached) gcc 2024-04-25T11:22:59.205 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-25T11:22:59.206 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T11:22:59.207 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-25T11:22:59.208 INFO:tasks.workunit.client.0.smithi136.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T11:22:59.335 INFO:tasks.workunit.client.0.smithi136.stdout:checking for main in -lm... yes 2024-04-25T11:22:59.450 INFO:tasks.workunit.client.0.smithi136.stdout:checking for main in -lpthread... yes 2024-04-25T11:22:59.782 INFO:tasks.workunit.client.0.smithi136.stdout:checking for ANSI C header files... yes 2024-04-25T11:22:59.874 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T11:22:59.979 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/types.h... yes 2024-04-25T11:23:00.098 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/stat.h... yes 2024-04-25T11:23:00.190 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdlib.h... yes 2024-04-25T11:23:00.311 INFO:tasks.workunit.client.0.smithi136.stdout:checking for string.h... yes 2024-04-25T11:23:00.405 INFO:tasks.workunit.client.0.smithi136.stdout:checking for memory.h... yes 2024-04-25T11:23:00.509 INFO:tasks.workunit.client.0.smithi136.stdout:checking for strings.h... yes 2024-04-25T11:23:00.600 INFO:tasks.workunit.client.0.smithi136.stdout:checking for inttypes.h... yes 2024-04-25T11:23:00.710 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdint.h... yes 2024-04-25T11:23:00.811 INFO:tasks.workunit.client.0.smithi136.stdout:checking for unistd.h... yes 2024-04-25T11:23:00.921 INFO:tasks.workunit.client.0.smithi136.stdout:checking pthread.h usability... yes 2024-04-25T11:23:00.964 INFO:tasks.workunit.client.0.smithi136.stdout:checking pthread.h presence... yes 2024-04-25T11:23:00.964 INFO:tasks.workunit.client.0.smithi136.stdout:checking for pthread.h... yes 2024-04-25T11:23:01.097 INFO:tasks.workunit.client.0.smithi136.stdout:checking fcntl.h usability... yes 2024-04-25T11:23:01.142 INFO:tasks.workunit.client.0.smithi136.stdout:checking fcntl.h presence... yes 2024-04-25T11:23:01.143 INFO:tasks.workunit.client.0.smithi136.stdout:checking for fcntl.h... yes 2024-04-25T11:23:01.273 INFO:tasks.workunit.client.0.smithi136.stdout:checking limits.h usability... yes 2024-04-25T11:23:01.306 INFO:tasks.workunit.client.0.smithi136.stdout:checking limits.h presence... yes 2024-04-25T11:23:01.307 INFO:tasks.workunit.client.0.smithi136.stdout:checking for limits.h... yes 2024-04-25T11:23:01.339 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdint.h... (cached) yes 2024-04-25T11:23:01.420 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/time.h usability... yes 2024-04-25T11:23:01.459 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/time.h presence... yes 2024-04-25T11:23:01.459 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/time.h... yes 2024-04-25T11:23:01.500 INFO:tasks.workunit.client.0.smithi136.stdout:checking for unistd.h... (cached) yes 2024-04-25T11:23:01.582 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/vfs.h usability... yes 2024-04-25T11:23:01.624 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/vfs.h presence... yes 2024-04-25T11:23:01.624 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/vfs.h... yes 2024-04-25T11:23:01.753 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/limits.h usability... no 2024-04-25T11:23:01.804 INFO:tasks.workunit.client.0.smithi136.stdout:checking sys/limits.h presence... no 2024-04-25T11:23:01.805 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/limits.h... no 2024-04-25T11:23:01.918 INFO:tasks.workunit.client.0.smithi136.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T11:23:01.975 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-25T11:23:01.979 INFO:tasks.workunit.client.0.smithi136.stdout:checking for function prototypes... yes 2024-04-25T11:23:02.035 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether setvbuf arguments are reversed... no 2024-04-25T11:23:02.142 INFO:tasks.workunit.client.0.smithi136.stdout:checking for system... yes 2024-04-25T11:23:02.290 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gettimeofday... yes 2024-04-25T11:23:02.406 INFO:tasks.workunit.client.0.smithi136.stdout:checking for mkdir... yes 2024-04-25T11:23:02.527 INFO:tasks.workunit.client.0.smithi136.stdout:checking for strerror... yes 2024-04-25T11:23:02.642 INFO:tasks.workunit.client.0.smithi136.stdout:checking for open64... yes 2024-04-25T11:23:02.757 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stat64... yes 2024-04-25T11:23:02.886 INFO:tasks.workunit.client.0.smithi136.stdout:checking for fseeko64... yes 2024-04-25T11:23:03.007 INFO:tasks.workunit.client.0.smithi136.stdout:checking for lrand48_r... yes 2024-04-25T11:23:03.125 INFO:tasks.workunit.client.0.smithi136.stdout:checking for srand48_r... yes 2024-04-25T11:23:03.179 INFO:tasks.workunit.client.0.smithi136.stdout:configure: creating ./config.status 2024-04-25T11:23:03.346 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: creating Makefile 2024-04-25T11:23:03.364 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: creating config.h 2024-04-25T11:23:03.393 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: executing depfiles commands 2024-04-25T11:23:03.484 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: executing default commands 2024-04-25T11:23:03.586 INFO:tasks.workunit.client.0.smithi136.stderr:+ make 2024-04-25T11:23:03.643 INFO:tasks.workunit.client.0.smithi136.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-25T11:23:03.649 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-25T11:23:03.650 INFO:tasks.workunit.client.0.smithi136.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-25T11:23:03.650 INFO:tasks.workunit.client.0.smithi136.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-25T11:23:03.650 INFO:tasks.workunit.client.0.smithi136.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-25T11:23:03.651 INFO:tasks.workunit.client.0.smithi136.stderr: any GNU archive site. 2024-04-25T11:23:03.656 INFO:tasks.workunit.client.0.smithi136.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-25T11:23:03.660 INFO:tasks.workunit.client.0.smithi136.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-25T11:23:03.661 INFO:tasks.workunit.client.0.smithi136.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-25T11:23:03.661 INFO:tasks.workunit.client.0.smithi136.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-25T11:23:03.661 INFO:tasks.workunit.client.0.smithi136.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-25T11:23:03.661 INFO:tasks.workunit.client.0.smithi136.stderr: Grab them from any GNU archive site. 2024-04-25T11:23:04.006 INFO:tasks.workunit.client.0.smithi136.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-25T11:23:04.161 INFO:tasks.workunit.client.0.smithi136.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T11:23:04.161 INFO:tasks.workunit.client.0.smithi136.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T11:23:04.161 INFO:tasks.workunit.client.0.smithi136.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T11:23:04.161 INFO:tasks.workunit.client.0.smithi136.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:4: You should run autoupdate. 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:4: the top level 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:6: You should run autoupdate. 2024-04-25T11:23:04.293 INFO:tasks.workunit.client.0.smithi136.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:6: the top level 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:10: You should run autoupdate. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:10: the top level 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:36: You should run autoupdate. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:36: the top level 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:42: You should run autoupdate. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:42: the top level 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-04-25T11:23:04.294 INFO:tasks.workunit.client.0.smithi136.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-04-25T11:23:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:45: the top level 2024-04-25T11:23:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-04-25T11:23:04.295 INFO:tasks.workunit.client.0.smithi136.stderr:configure.in:50: You should run autoupdate. 2024-04-25T11:23:04.373 INFO:tasks.workunit.client.0.smithi136.stdout:/bin/bash ./config.status --recheck 2024-04-25T11:23:04.460 INFO:tasks.workunit.client.0.smithi136.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-04-25T11:23:04.687 INFO:tasks.workunit.client.0.smithi136.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T11:23:05.706 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether build environment is sane... yes 2024-04-25T11:23:05.718 INFO:tasks.workunit.client.0.smithi136.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-04-25T11:23:05.719 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gawk... gawk 2024-04-25T11:23:05.761 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T11:23:05.817 INFO:tasks.workunit.client.0.smithi136.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T11:23:05.818 INFO:tasks.workunit.client.0.smithi136.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T11:23:05.856 INFO:tasks.workunit.client.0.smithi136.stdout:checking for style of include used by make... GNU 2024-04-25T11:23:05.867 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc... gcc 2024-04-25T11:23:06.116 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether the C compiler works... yes 2024-04-25T11:23:06.116 INFO:tasks.workunit.client.0.smithi136.stdout:checking for C compiler default output file name... a.out 2024-04-25T11:23:06.192 INFO:tasks.workunit.client.0.smithi136.stdout:checking for suffix of executables... 2024-04-25T11:23:06.276 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether we are cross compiling... no 2024-04-25T11:23:06.356 INFO:tasks.workunit.client.0.smithi136.stdout:checking for suffix of object files... o 2024-04-25T11:23:06.404 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether the compiler supports GNU C... yes 2024-04-25T11:23:06.452 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether gcc accepts -g... yes 2024-04-25T11:23:06.544 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc option to enable C11 features... none needed 2024-04-25T11:23:06.863 INFO:tasks.workunit.client.0.smithi136.stdout:checking dependency style of gcc... gcc3 2024-04-25T11:23:06.931 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdio.h... yes 2024-04-25T11:23:07.009 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdlib.h... yes 2024-04-25T11:23:07.068 INFO:tasks.workunit.client.0.smithi136.stdout:checking for string.h... yes 2024-04-25T11:23:07.122 INFO:tasks.workunit.client.0.smithi136.stdout:checking for inttypes.h... yes 2024-04-25T11:23:07.179 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdint.h... yes 2024-04-25T11:23:07.237 INFO:tasks.workunit.client.0.smithi136.stdout:checking for strings.h... yes 2024-04-25T11:23:07.292 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/stat.h... yes 2024-04-25T11:23:07.346 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/types.h... yes 2024-04-25T11:23:07.404 INFO:tasks.workunit.client.0.smithi136.stdout:checking for unistd.h... yes 2024-04-25T11:23:07.469 INFO:tasks.workunit.client.0.smithi136.stdout:checking for wchar.h... yes 2024-04-25T11:23:07.545 INFO:tasks.workunit.client.0.smithi136.stdout:checking for minix/config.h... no 2024-04-25T11:23:07.600 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/time.h... yes 2024-04-25T11:23:07.666 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-25T11:23:07.715 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-04-25T11:23:07.726 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc... (cached) gcc 2024-04-25T11:23:07.836 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-04-25T11:23:07.836 INFO:tasks.workunit.client.0.smithi136.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T11:23:07.837 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-04-25T11:23:07.837 INFO:tasks.workunit.client.0.smithi136.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T11:23:07.910 INFO:tasks.workunit.client.0.smithi136.stdout:checking for main in -lm... yes 2024-04-25T11:23:07.989 INFO:tasks.workunit.client.0.smithi136.stdout:checking for main in -lpthread... yes 2024-04-25T11:23:07.994 INFO:tasks.workunit.client.0.smithi136.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T11:23:07.995 INFO:tasks.workunit.client.0.smithi136.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T11:23:08.047 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T11:23:08.107 INFO:tasks.workunit.client.0.smithi136.stdout:checking for pthread.h... yes 2024-04-25T11:23:08.169 INFO:tasks.workunit.client.0.smithi136.stdout:checking for fcntl.h... yes 2024-04-25T11:23:08.230 INFO:tasks.workunit.client.0.smithi136.stdout:checking for limits.h... yes 2024-04-25T11:23:08.233 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stdint.h... (cached) yes 2024-04-25T11:23:08.234 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/time.h... (cached) yes 2024-04-25T11:23:08.235 INFO:tasks.workunit.client.0.smithi136.stdout:checking for unistd.h... (cached) yes 2024-04-25T11:23:08.334 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/vfs.h... yes 2024-04-25T11:23:08.423 INFO:tasks.workunit.client.0.smithi136.stdout:checking for sys/limits.h... no 2024-04-25T11:23:08.466 INFO:tasks.workunit.client.0.smithi136.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T11:23:08.557 INFO:tasks.workunit.client.0.smithi136.stdout:checking for system... yes 2024-04-25T11:23:08.641 INFO:tasks.workunit.client.0.smithi136.stdout:checking for gettimeofday... yes 2024-04-25T11:23:08.721 INFO:tasks.workunit.client.0.smithi136.stdout:checking for mkdir... yes 2024-04-25T11:23:08.802 INFO:tasks.workunit.client.0.smithi136.stdout:checking for strerror... yes 2024-04-25T11:23:08.883 INFO:tasks.workunit.client.0.smithi136.stdout:checking for open64... yes 2024-04-25T11:23:08.968 INFO:tasks.workunit.client.0.smithi136.stdout:checking for stat64... yes 2024-04-25T11:23:09.051 INFO:tasks.workunit.client.0.smithi136.stdout:checking for fseeko64... yes 2024-04-25T11:23:09.138 INFO:tasks.workunit.client.0.smithi136.stdout:checking for lrand48_r... yes 2024-04-25T11:23:09.218 INFO:tasks.workunit.client.0.smithi136.stdout:checking for srand48_r... yes 2024-04-25T11:23:09.251 INFO:tasks.workunit.client.0.smithi136.stdout:configure: creating ./config.status 2024-04-25T11:23:09.400 INFO:tasks.workunit.client.0.smithi136.stdout: /bin/bash ./config.status 2024-04-25T11:23:09.479 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: creating Makefile 2024-04-25T11:23:09.499 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: creating config.h 2024-04-25T11:23:09.539 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: executing depfiles commands 2024-04-25T11:23:09.557 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: executing default commands 2024-04-25T11:23:09.670 INFO:tasks.workunit.client.0.smithi136.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-25T11:23:09.733 INFO:tasks.workunit.client.0.smithi136.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-04-25T11:23:09.818 INFO:tasks.workunit.client.0.smithi136.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T11:23:09.818 INFO:tasks.workunit.client.0.smithi136.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T11:23:09.818 INFO:tasks.workunit.client.0.smithi136.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T11:23:09.818 INFO:tasks.workunit.client.0.smithi136.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T11:23:09.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:09.971 INFO:tasks.workunit.client.0.smithi136.stdout:rm -f stamp-h1 2024-04-25T11:23:09.978 INFO:tasks.workunit.client.0.smithi136.stdout:touch config.h.in 2024-04-25T11:23:09.985 INFO:tasks.workunit.client.0.smithi136.stdout:cd . && /bin/bash ./config.status config.h 2024-04-25T11:23:10.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.073+0000 7f144cc2c640 1 Processor -- start 2024-04-25T11:23:10.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.073+0000 7f144cc2c640 1 -- start start 2024-04-25T11:23:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144cc2c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f14480ca720 0x7f14480ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144cc2c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14480cbad0 con 0x7f14480c8cd0 2024-04-25T11:23:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144cc2c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14480c7c20 con 0x7f14480ce260 2024-04-25T11:23:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144cc2c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f14480c8550 con 0x7f14480ca720 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f14480c8cd0 0x7f144814f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39818/0 (socket says 172.21.15.136:39818) 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144659c640 1 -- 172.21.15.136:0/1494937454 learned_addr learned my addr 172.21.15.136:0/1494937454 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f1446d9d640 1 --2- 172.21.15.136:0/1494937454 >> v2:172.21.15.136:3300/0 conn(0x7f14480ca720 0x7f14480ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144559a640 1 -- 172.21.15.136:0/1494937454 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258743175 0 0) 0x7f14480cbad0 con 0x7f14480c8cd0 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f144559a640 1 -- 172.21.15.136:0/1494937454 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1430003530 con 0x7f14480c8cd0 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f1446d9d640 1 -- 172.21.15.136:0/1494937454 >> v1:172.21.15.157:6789/0 conn(0x7f14480ce260 legacy=0x7f14480ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f1446d9d640 1 -- 172.21.15.136:0/1494937454 >> v1:172.21.15.136:6789/0 conn(0x7f14480c8cd0 legacy=0x7f144814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.077+0000 7f1446d9d640 1 -- 172.21.15.136:0/1494937454 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14480c7f00 con 0x7f14480ca720 2024-04-25T11:23:10.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f1446d9d640 1 --2- 172.21.15.136:0/1494937454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f14480ce670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1440009d90 tx=0x7f144001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb55218e9705db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144559a640 1 -- 172.21.15.136:0/1494937454 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14400b4070 con 0x7f14480ca720 2024-04-25T11:23:10.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144559a640 1 -- 172.21.15.136:0/1494937454 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14400af600 con 0x7f14480ca720 2024-04-25T11:23:10.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144559a640 1 -- 172.21.15.136:0/1494937454 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14400af9d0 con 0x7f14480ca720 2024-04-25T11:23:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- 172.21.15.136:0/1494937454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 msgr2=0x7f14480ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 --2- 172.21.15.136:0/1494937454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f14480ce670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1440009d90 tx=0x7f144001f150 comp rx=0 tx=0).stop 2024-04-25T11:23:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- 172.21.15.136:0/1494937454 shutdown_connections 2024-04-25T11:23:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 --2- 172.21.15.136:0/1494937454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f14480ce670 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- 172.21.15.136:0/1494937454 >> 172.21.15.136:0/1494937454 conn(0x7f144806f9d0 msgr2=0x7f144806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:10.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- 172.21.15.136:0/1494937454 shutdown_connections 2024-04-25T11:23:10.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- 172.21.15.136:0/1494937454 wait complete. 2024-04-25T11:23:10.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 Processor -- start 2024-04-25T11:23:10.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.081+0000 7f144cc2c640 1 -- start start 2024-04-25T11:23:10.078 INFO:tasks.workunit.client.0.smithi136.stdout:config.status: creating config.h 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 0x7f14481e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14480ce260 0x7f1448148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f14480cb720 con 0x7f14480ca720 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14480c91e0 con 0x7f14480c8cd0 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1448159640 con 0x7f14480ce260 2024-04-25T11:23:10.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35070/0 (socket says 172.21.15.136:35070) 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 -- 172.21.15.136:0/2222730287 learned_addr learned my addr 172.21.15.136:0/2222730287 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1446d9d640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14480ce260 0x7f1448148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144659c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 0x7f14481e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 -- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14480ce260 msgr2=0x7f1448148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14480ce260 0x7f1448148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 -- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 msgr2=0x7f14481e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 0x7f14481e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 -- 172.21.15.136:0/2222730287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1448159170 con 0x7f14480ca720 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144659c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 0x7f14481e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1445d9b640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f143c00c3c0 tx=0x7f143c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c017070 con 0x7f14480ca720 2024-04-25T11:23:10.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f143c0052d0 con 0x7f14480ca720 2024-04-25T11:23:10.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1448153440 con 0x7f14480ca720 2024-04-25T11:23:10.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c003840 con 0x7f14480ca720 2024-04-25T11:23:10.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1448158150 con 0x7f14480ca720 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1448156e20 con 0x7f14480ca720 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f143c01a070 con 0x7f14480ca720 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.085+0000 7f1436ffd640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 0x7f1428071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.089+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f143c08e8c0 con 0x7f14480ca720 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.089+0000 7f144659c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 0x7f1428071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.089+0000 7f144659c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 0x7f1428071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f14380011f0 tx=0x7f1438009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.093+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f143c005480 con 0x7f14480ca720 2024-04-25T11:23:10.137 INFO:tasks.workunit.client.0.smithi136.stdout:make all-am 2024-04-25T11:23:10.172 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T11:23:10.172 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-25T11:23:10.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.309+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f144806a350 con 0x7f14480ca720 2024-04-25T11:23:10.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.309+0000 7f1436ffd640 1 -- 172.21.15.136:0/2222730287 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f143c063430 con 0x7f14480ca720 2024-04-25T11:23:10.306 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:10.306 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:10.307 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:10.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 msgr2=0x7f1428071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 0x7f1428071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f14380011f0 tx=0x7f1438009040 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 msgr2=0x7f1448158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f143c00c3c0 tx=0x7f143c00c880 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 shutdown_connections 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f142806f400 0x7f1428071910 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14480ce260 0x7f1448148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14480ca720 0x7f1448158c30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 --2- 172.21.15.136:0/2222730287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14480c8cd0 0x7f14481e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 >> 172.21.15.136:0/2222730287 conn(0x7f144806f9d0 msgr2=0x7f144806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:10.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 shutdown_connections 2024-04-25T11:23:10.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.313+0000 7f144cc2c640 1 -- 172.21.15.136:0/2222730287 wait complete. 2024-04-25T11:23:10.339 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=4) 2024-04-25T11:23:10.339 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T11:23:10.339 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:23:10.380 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-25T11:23:10.390 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-25T11:23:10.393 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T11:23:10.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:10.496 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-25T11:23:10.508 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-25T11:23:10.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 Processor -- start 2024-04-25T11:23:10.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 -- start start 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c3c0ce260 0x7f9c3c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c0cbad0 con 0x7f9c3c0ca010 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c0c8550 con 0x7f9c3c0ce260 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c40c7b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c0c7c20 con 0x7f9c3c0c8cd0 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c3a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.577+0000 7f9c39d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c3c0ce260 0x7f9c3c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:41316/0 (socket says 172.21.15.136:41316) 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c3c0ce260 0x7f9c3c0ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:49074/0 (socket says 172.21.15.136:49074) 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3ad9d640 1 --1- 172.21.15.136:0/746911038 >> v1:172.21.15.157:6789/0 conn(0x7f9c3c0ca010 0x7f9c3c0ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:37890/0 (socket says 172.21.15.136:37890) 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3a59c640 1 -- 172.21.15.136:0/746911038 learned_addr learned my addr 172.21.15.136:0/746911038 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 -- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 msgr2=0x7f9c3c14f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 -- 172.21.15.136:0/746911038 >> v1:172.21.15.157:6789/0 conn(0x7f9c3c0ca010 legacy=0x7f9c3c0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 -- 172.21.15.136:0/746911038 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c0c7f00 con 0x7f9c3c0ce260 2024-04-25T11:23:10.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3a59c640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:23:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c39d9b640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c0ce670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0c86d0 tx=0x7f9c2c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda2a81dcd0f6681 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3959a640 1 -- 172.21.15.136:0/746911038 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0b5070 con 0x7f9c3c0ce260 2024-04-25T11:23:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3959a640 1 -- 172.21.15.136:0/746911038 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2c0a8990 con 0x7f9c3c0ce260 2024-04-25T11:23:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c3959a640 1 -- 172.21.15.136:0/746911038 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c0a8d60 con 0x7f9c3c0ce260 2024-04-25T11:23:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c40c7b640 1 -- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0ce260 msgr2=0x7f9c3c0ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.581+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c0ce670 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0c86d0 tx=0x7f9c2c01ec90 comp rx=0 tx=0).stop 2024-04-25T11:23:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- 172.21.15.136:0/746911038 shutdown_connections 2024-04-25T11:23:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c0ce670 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/746911038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0c8cd0 0x7f9c3c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- 172.21.15.136:0/746911038 >> 172.21.15.136:0/746911038 conn(0x7f9c3c06f9d0 msgr2=0x7f9c3c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:10.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- 172.21.15.136:0/746911038 shutdown_connections 2024-04-25T11:23:10.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- 172.21.15.136:0/746911038 wait complete. 2024-04-25T11:23:10.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 Processor -- start 2024-04-25T11:23:10.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- start start 2024-04-25T11:23:10.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 0x7f9c3c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0ca010 0x7f9c3c1e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c0cb720 con 0x7f9c3c0ce260 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c14fbc0 con 0x7f9c3c0c8cd0 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c40c7b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c14d4a0 con 0x7f9c3c0ca010 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35080/0 (socket says 172.21.15.136:35080) 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 -- 172.21.15.136:0/4167282643 learned_addr learned my addr 172.21.15.136:0/4167282643 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c39d9b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0ca010 0x7f9c3c1e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3a59c640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 0x7f9c3c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 -- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0ca010 msgr2=0x7f9c3c1e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0ca010 0x7f9c3c1e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 -- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 msgr2=0x7f9c3c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 0x7f9c3c0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 -- 172.21.15.136:0/4167282643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c0cd680 con 0x7f9c3c0ce260 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3a59c640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 0x7f9c3c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:10.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.585+0000 7f9c3ad9d640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c6f0 tx=0x7f9c3400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34017070 con 0x7f9c3c0ce260 2024-04-25T11:23:10.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c34005db0 con 0x7f9c3c0ce260 2024-04-25T11:23:10.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400a840 con 0x7f9c3c0ce260 2024-04-25T11:23:10.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c0cd960 con 0x7f9c3c0ce260 2024-04-25T11:23:10.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c0cc490 con 0x7f9c3c0ce260 2024-04-25T11:23:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c06a350 con 0x7f9c3c0ce260 2024-04-25T11:23:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9c3401a070 con 0x7f9c3c0ce260 2024-04-25T11:23:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 0x7f9c1c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9c3408ef70 con 0x7f9c3c0ce260 2024-04-25T11:23:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c3a59c640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 0x7f9c1c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:10.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.589+0000 7f9c3a59c640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 0x7f9c1c071940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009b00 tx=0x7f9c30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:10.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.593+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c3405b4f0 con 0x7f9c3c0ce260 2024-04-25T11:23:10.635 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-25T11:23:10.646 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-25T11:23:10.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.789+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c3c075f80 con 0x7f9c3c0ce260 2024-04-25T11:23:10.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.789+0000 7f9c2affd640 1 -- 172.21.15.136:0/4167282643 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f9c3405f7f0 con 0x7f9c3c0ce260 2024-04-25T11:23:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:10.785 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:10.785 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:10.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 msgr2=0x7f9c1c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 0x7f9c1c071940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009b00 tx=0x7f9c30008040 comp rx=0 tx=0).stop 2024-04-25T11:23:10.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 msgr2=0x7f9c3c148660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:10.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400c6f0 tx=0x7f9c3400cbb0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.787 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-25T11:23:10.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 shutdown_connections 2024-04-25T11:23:10.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c3c0ce260 0x7f9c3c148660 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9c1c06f430 0x7f9c1c071940 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c3c0ca010 0x7f9c3c1e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 --2- 172.21.15.136:0/4167282643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c3c0c8cd0 0x7f9c3c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:10.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 >> 172.21.15.136:0/4167282643 conn(0x7f9c3c06f9d0 msgr2=0x7f9c3c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:10.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 shutdown_connections 2024-04-25T11:23:10.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:10.793+0000 7f9c40c7b640 1 -- 172.21.15.136:0/4167282643 wait complete. 2024-04-25T11:23:10.798 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-25T11:23:10.805 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:10.961 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-25T11:23:10.974 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-25T11:23:11.066 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-25T11:23:11.077 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-25T11:23:11.139 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-25T11:23:11.150 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-25T11:23:11.226 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-25T11:23:11.238 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-25T11:23:11.484 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-25T11:23:11.496 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-25T11:23:11.692 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-25T11:23:11.705 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-25T11:23:11.942 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-25T11:23:11.957 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-25T11:23:12.071 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-25T11:23:12.082 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-25T11:23:12.136 INFO:tasks.workunit.client.0.smithi136.stderr:ffsb_op.c: In function 'print_op_results': 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~ 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr: 86 | sprintf(buf, "NA\0"); 2024-04-25T11:23:12.137 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~ 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr:ffsb_op.c: In function 'print_results': 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~ 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr:In file included from /usr/include/stdio.h:894, 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: from ffsb_op.c:20: 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.152 INFO:tasks.workunit.client.0.smithi136.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-04-25T11:23:12.153 INFO:tasks.workunit.client.0.smithi136.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.153 INFO:tasks.workunit.client.0.smithi136.stderr: 40 | __va_arg_pack ()); 2024-04-25T11:23:12.153 INFO:tasks.workunit.client.0.smithi136.stderr: | ~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.202 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-25T11:23:12.214 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-25T11:23:12.395 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-25T11:23:12.407 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-25T11:23:12.471 INFO:tasks.workunit.client.0.smithi136.stderr:In file included from parser.h:22, 2024-04-25T11:23:12.471 INFO:tasks.workunit.client.0.smithi136.stderr: from parser.c:25: 2024-04-25T11:23:12.471 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c: In function 'init_filesys': 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.473 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.474 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.475 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.476 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c: In function 'init_tg_stats': 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.477 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.478 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.479 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr: | ^ 2024-04-25T11:23:12.480 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~ 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: 36 | container_of(ptr, type, member) 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~ 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~ 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T11:23:12.481 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c: In function 'get_optstr': 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr:parser.c:200:14: note: source object declared here 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr: 200 | char temp[BUFSIZE]; 2024-04-25T11:23:12.527 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~ 2024-04-25T11:23:12.544 INFO:tasks.workunit.client.0.smithi136.stderr:In file included from /usr/include/string.h:535, 2024-04-25T11:23:12.544 INFO:tasks.workunit.client.0.smithi136.stderr: from parser.c:19: 2024-04-25T11:23:12.544 INFO:tasks.workunit.client.0.smithi136.stderr:In function 'memcpy', 2024-04-25T11:23:12.544 INFO:tasks.workunit.client.0.smithi136.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-04-25T11:23:12.544 INFO:tasks.workunit.client.0.smithi136.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-25T11:23:12.545 INFO:tasks.workunit.client.0.smithi136.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-04-25T11:23:12.545 INFO:tasks.workunit.client.0.smithi136.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.545 INFO:tasks.workunit.client.0.smithi136.stderr: 30 | __glibc_objsize0 (__dest)); 2024-04-25T11:23:12.545 INFO:tasks.workunit.client.0.smithi136.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T11:23:12.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:12.874 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-25T11:23:12.886 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-25T11:23:12.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 Processor -- start 2024-04-25T11:23:12.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 -- start start 2024-04-25T11:23:12.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6fd814f840 0x7f6fd8074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd8073200 0x7f6fd8073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd80c3750 con 0x7f6fd80c3c30 2024-04-25T11:23:12.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f6fd806dc00 con 0x7f6fd814f840 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fddeef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd814fc50 con 0x7f6fd8073200 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6fd814f840 0x7f6fd8074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6fd80c3c30 0x7f6fd80c4040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39828/0 (socket says 172.21.15.136:39828) 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd7fff640 1 -- 172.21.15.136:0/10530769 learned_addr learned my addr 172.21.15.136:0/10530769 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd77fe640 1 -- 172.21.15.136:0/10530769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd8073200 msgr2=0x7f6fd8073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd77fe640 1 --2- 172.21.15.136:0/10530769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd8073200 0x7f6fd8073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd77fe640 1 -- 172.21.15.136:0/10530769 >> v1:172.21.15.136:6789/0 conn(0x7f6fd80c3c30 legacy=0x7f6fd80c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd77fe640 1 -- 172.21.15.136:0/10530769 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd80c3190 con 0x7f6fd814f840 2024-04-25T11:23:12.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.969+0000 7f6fd67fc640 1 -- 172.21.15.136:0/10530769 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931721352 0 0) 0x7f6fd80c3750 con 0x7f6fd80c3c30 2024-04-25T11:23:12.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fd77fe640 1 --2- 172.21.15.136:0/10530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd814f840 0x7f6fd8074980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00ba60 tx=0x7f6fcc01eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810c51de9b79e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:12.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fd67fc640 1 -- 172.21.15.136:0/10530769 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0b4070 con 0x7f6fd814f840 2024-04-25T11:23:12.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fd67fc640 1 -- 172.21.15.136:0/10530769 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0a7940 con 0x7f6fd814f840 2024-04-25T11:23:12.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fd67fc640 1 -- 172.21.15.136:0/10530769 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0a7d10 con 0x7f6fd814f840 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 -- 172.21.15.136:0/10530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd814f840 msgr2=0x7f6fd8074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 --2- 172.21.15.136:0/10530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd814f840 0x7f6fd8074980 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00ba60 tx=0x7f6fcc01eb70 comp rx=0 tx=0).stop 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 -- 172.21.15.136:0/10530769 shutdown_connections 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 --2- 172.21.15.136:0/10530769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd8073200 0x7f6fd8073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 --2- 172.21.15.136:0/10530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd814f840 0x7f6fd8074980 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 -- 172.21.15.136:0/10530769 >> 172.21.15.136:0/10530769 conn(0x7f6fd80ca5a0 msgr2=0x7f6fd80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:12.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 -- 172.21.15.136:0/10530769 shutdown_connections 2024-04-25T11:23:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.973+0000 7f6fddeef640 1 -- 172.21.15.136:0/10530769 wait complete. 2024-04-25T11:23:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 Processor -- start 2024-04-25T11:23:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 -- start start 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fd80c3c30 0x7f6fd80c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd814f840 0x7f6fd814b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81597f0 con 0x7f6fd8073200 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8153380 con 0x7f6fd80c3c30 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd8159970 con 0x7f6fd814f840 2024-04-25T11:23:12.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35086/0 (socket says 172.21.15.136:35086) 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 -- 172.21.15.136:0/2797652728 learned_addr learned my addr 172.21.15.136:0/2797652728 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd7fff640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd814f840 0x7f6fd814b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 -- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd814f840 msgr2=0x7f6fd814b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd6ffd640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fd80c3c30 0x7f6fd80c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd814f840 0x7f6fd814b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 -- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fd80c3c30 msgr2=0x7f6fd80c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fd80c3c30 0x7f6fd80c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 -- 172.21.15.136:0/2797652728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd815a0c0 con 0x7f6fd8073200 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fd77fe640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc01c7a0 tx=0x7f6fcc01b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0b4040 con 0x7f6fd8073200 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0bd070 con 0x7f6fd8073200 2024-04-25T11:23:12.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0afa00 con 0x7f6fd8073200 2024-04-25T11:23:12.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.977+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd8156430 con 0x7f6fd8073200 2024-04-25T11:23:12.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd8154a30 con 0x7f6fd8073200 2024-04-25T11:23:12.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd806a350 con 0x7f6fd8073200 2024-04-25T11:23:12.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6fcc0afba0 con 0x7f6fd8073200 2024-04-25T11:23:12.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fbffff640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 0x7f6fc4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:12.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6fcc129cd0 con 0x7f6fd8073200 2024-04-25T11:23:12.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fd6ffd640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 0x7f6fc4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:12.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.981+0000 7f6fd6ffd640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 0x7f6fc4071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81591c0 tx=0x7f6fc8007310 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:12.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:12.985+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6fcc018760 con 0x7f6fd8073200 2024-04-25T11:23:12.987 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-25T11:23:12.999 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-25T11:23:13.124 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-25T11:23:13.135 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-25T11:23:13.184 INFO:tasks.workunit.client.0.smithi136.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-25T11:23:13.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fd80c2ed0 con 0x7f6fd8073200 2024-04-25T11:23:13.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fbffff640 1 -- 172.21.15.136:0/2797652728 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f6fcc0f67c0 con 0x7f6fd8073200 2024-04-25T11:23:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:13.192 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:13.192 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 msgr2=0x7f6fc4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 0x7f6fc4071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81591c0 tx=0x7f6fc8007310 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 msgr2=0x7f6fd81f5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc01c7a0 tx=0x7f6fcc01b910 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 shutdown_connections 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6fc406f400 0x7f6fc4071910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fd814f840 0x7f6fd814b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fd80c3c30 0x7f6fd80c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 --2- 172.21.15.136:0/2797652728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fd8073200 0x7f6fd81f5c50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 >> 172.21.15.136:0/2797652728 conn(0x7f6fd80ca5a0 msgr2=0x7f6fd814c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:13.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 shutdown_connections 2024-04-25T11:23:13.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:13.197+0000 7f6fddeef640 1 -- 172.21.15.136:0/2797652728 wait complete. 2024-04-25T11:23:13.199 INFO:tasks.workunit.client.0.smithi136.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-25T11:23:13.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:13.327 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T11:23:13.328 INFO:tasks.workunit.client.0.smithi136.stderr:+ cd .. 2024-04-25T11:23:13.330 INFO:tasks.workunit.client.0.smithi136.stderr:+ mkdir tmp 2024-04-25T11:23:13.334 INFO:tasks.workunit.client.0.smithi136.stderr:+ cd tmp 2024-04-25T11:23:13.336 INFO:tasks.workunit.client.0.smithi136.stderr:+ for f in $mydir/*.ffsb 2024-04-25T11:23:13.336 INFO:tasks.workunit.client.0.smithi136.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-25T11:23:13.337 INFO:tasks.workunit.client.0.smithi136.stdout:FFSB version 6.0-RC2 started 2024-04-25T11:23:13.337 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout:benchmark time = 300 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout:ThreadGroup 0 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout:================ 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: num_threads = 32 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: read_random = off 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: read_size = 0 (0B) 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: read_blocksize = 0 (0B) 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: read_skip = off 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: read_skipsize = 0 (0B) 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: write_random = on 2024-04-25T11:23:13.341 INFO:tasks.workunit.client.0.smithi136.stdout: write_size = 5242880 (5MB) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: fsync_file = 0 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: write_blocksize = 4096 (4KB) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: wait time = 0 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: op weights 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: read = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: readall = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: write = 1 (100.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: create = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: append = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: delete = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: metaop = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: createdir = 0 (0.00%) 2024-04-25T11:23:13.342 INFO:tasks.workunit.client.0.smithi136.stdout: stat = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: writeall = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: writeall_fsync = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: open_close = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: write_fsync = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: create_fsync = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: append_fsync = 0 (0.00%) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout:FileSystem . 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout:========== 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: num_dirs = 0 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: starting files = 128 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: min file size = 104857600 (100MB) 2024-04-25T11:23:13.343 INFO:tasks.workunit.client.0.smithi136.stdout: max file size = 104857600 (100MB) 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: directio = off 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: alignedio = on 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: bufferedio = off 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: aging is off 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: current utilization = 0.04% 2024-04-25T11:23:13.344 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:23:13.345 INFO:tasks.workunit.client.0.smithi136.stdout:checking existing fs: . 2024-04-25T11:23:13.345 INFO:tasks.workunit.client.0.smithi136.stdout:recreating new fileset 2024-04-25T11:23:13.345 INFO:tasks.workunit.client.0.smithi136.stderr:opendir: No such file or directory 2024-04-25T11:23:15.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:15.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 Processor -- start 2024-04-25T11:23:15.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 -- start start 2024-04-25T11:23:15.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 0x7f38c80c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38c815ab40 0x7f38c80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c815d750 con 0x7f38c80ca290 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f38c815da30 con 0x7f38c815ab40 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38ce4e1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f38c815dd10 con 0x7f38c80b5060 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38c815ab40 0x7f38c80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38c815ab40 0x7f38c80ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38932/0 (socket says 172.21.15.136:38932) 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 -- 172.21.15.136:0/2369040239 learned_addr learned my addr 172.21.15.136:0/2369040239 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38c77fe640 1 --1- 172.21.15.136:0/2369040239 >> v1:172.21.15.136:6789/0 conn(0x7f38c80ca290 0x7f38c815a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48246/0 (socket says 172.21.15.136:48246) 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38c7fff640 1 --2- 172.21.15.136:0/2369040239 >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 0x7f38c80c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 -- 172.21.15.136:0/2369040239 >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 msgr2=0x7f38c80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 --2- 172.21.15.136:0/2369040239 >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 0x7f38c80c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 -- 172.21.15.136:0/2369040239 >> v1:172.21.15.136:6789/0 conn(0x7f38c80ca290 legacy=0x7f38c815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38c6ffd640 1 -- 172.21.15.136:0/2369040239 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727782721 0 0) 0x7f38c815d750 con 0x7f38c80ca290 2024-04-25T11:23:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38cca7e640 1 -- 172.21.15.136:0/2369040239 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c815dff0 con 0x7f38c815ab40 2024-04-25T11:23:15.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.393+0000 7f38c7fff640 1 --2- 172.21.15.136:0/2369040239 >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 0x7f38c80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:15.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38cca7e640 1 --2- 172.21.15.136:0/2369040239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c815ab40 0x7f38c80ca6a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f38c8068ff0 tx=0x7f38c001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a0ae7af9bcbefc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:15.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38c6ffd640 1 -- 172.21.15.136:0/2369040239 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c00b4070 con 0x7f38c815ab40 2024-04-25T11:23:15.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38c6ffd640 1 -- 172.21.15.136:0/2369040239 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c00af400 con 0x7f38c815ab40 2024-04-25T11:23:15.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38c6ffd640 1 -- 172.21.15.136:0/2369040239 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c00af7d0 con 0x7f38c815ab40 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2369040239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c815ab40 msgr2=0x7f38c80ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2369040239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c815ab40 0x7f38c80ca6a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f38c8068ff0 tx=0x7f38c001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2369040239 shutdown_connections 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2369040239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c815ab40 0x7f38c80ca6a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2369040239 >> v2:172.21.15.157:3300/0 conn(0x7f38c80b5060 0x7f38c80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2369040239 >> 172.21.15.136:0/2369040239 conn(0x7f38c8070f70 msgr2=0x7f38c8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2369040239 shutdown_connections 2024-04-25T11:23:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2369040239 wait complete. 2024-04-25T11:23:15.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 Processor -- start 2024-04-25T11:23:15.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.397+0000 7f38ce4e1640 1 -- start start 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38c80ca290 0x7f38c8149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38c815ab40 0x7f38c80b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f38c815d3f0 con 0x7f38c80b5060 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f38c8159f10 con 0x7f38c815ab40 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f38c80c4a60 con 0x7f38c80ca290 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38946/0 (socket says 172.21.15.136:38946) 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 -- 172.21.15.136:0/2174457577 learned_addr learned my addr 172.21.15.136:0/2174457577 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c77fe640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38c80ca290 0x7f38c8149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38cca7e640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38c815ab40 0x7f38c80b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 -- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38c80ca290 msgr2=0x7f38c8149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38c80ca290 0x7f38c8149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 -- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38c815ab40 msgr2=0x7f38c80b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38c815ab40 0x7f38c80b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 -- 172.21.15.136:0/2174457577 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c80ce610 con 0x7f38c80b5060 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c7fff640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c7f0 tx=0x7f38b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8015070 con 0x7f38c80b5060 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b8007820 con 0x7f38c80b5060 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c81f00a0 con 0x7f38c80b5060 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8007c10 con 0x7f38c80b5060 2024-04-25T11:23:15.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c81f0640 con 0x7f38c80b5060 2024-04-25T11:23:15.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.401+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c80cdd10 con 0x7f38c80b5060 2024-04-25T11:23:15.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.405+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f38b8018070 con 0x7f38c80b5060 2024-04-25T11:23:15.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.405+0000 7f38c4ff9640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 0x7f38b4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:15.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.405+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f38b808f6b0 con 0x7f38c80b5060 2024-04-25T11:23:15.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.405+0000 7f38c77fe640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 0x7f38b4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:15.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.405+0000 7f38c77fe640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 0x7f38b4071940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f38c80cc110 tx=0x7f38bc007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:15.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.409+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f38b805c2c0 con 0x7f38c80b5060 2024-04-25T11:23:15.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.641+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38c806a350 con 0x7f38c80b5060 2024-04-25T11:23:15.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.641+0000 7f38c4ff9640 1 -- 172.21.15.136:0/2174457577 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f38b805bbc0 con 0x7f38c80b5060 2024-04-25T11:23:15.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:15.637 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:15.637 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 msgr2=0x7f38b4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 0x7f38b4071940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f38c80cc110 tx=0x7f38bc007c10 comp rx=0 tx=0).stop 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 msgr2=0x7f38c80ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c7f0 tx=0x7f38b800ccb0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 shutdown_connections 2024-04-25T11:23:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38c815ab40 0x7f38c80b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f38b406f430 0x7f38b4071940 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38c80ca290 0x7f38c8149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 --2- 172.21.15.136:0/2174457577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38c80b5060 0x7f38c80ce0d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 >> 172.21.15.136:0/2174457577 conn(0x7f38c8070f70 msgr2=0x7f38c80cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 shutdown_connections 2024-04-25T11:23:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:15.645+0000 7f38ce4e1640 1 -- 172.21.15.136:0/2174457577 wait complete. 2024-04-25T11:23:15.665 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:17.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:17.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 Processor -- start 2024-04-25T11:23:17.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 -- start start 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4ee015a5f0 0x7f4ee00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee00c3880 con 0x7f4ee015aa00 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ee006d9b0 con 0x7f4ee0154060 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee63a4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4ee00c8fd0 con 0x7f4ee015a5f0 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee4941640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4ee015aa00 0x7f4ee0164700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48262/0 (socket says 172.21.15.136:48262) 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.817+0000 7f4ee4941640 1 -- 172.21.15.136:0/4120928488 learned_addr learned my addr 172.21.15.136:0/4120928488 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edf7fe640 1 --2- 172.21.15.136:0/4120928488 >> v2:172.21.15.157:3300/0 conn(0x7f4ee015a5f0 0x7f4ee00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2857430778 0 0) 0x7f4ee00c3880 con 0x7f4ee015aa00 2024-04-25T11:23:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ecc003530 con 0x7f4ee015aa00 2024-04-25T11:23:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3180215231 0 0) 0x7f4ecc003530 con 0x7f4ee015aa00 2024-04-25T11:23:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 >> v2:172.21.15.157:3300/0 conn(0x7f4ee015a5f0 msgr2=0x7f4ee00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 --2- 172.21.15.136:0/4120928488 >> v2:172.21.15.157:3300/0 conn(0x7f4ee015a5f0 0x7f4ee00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 >> v1:172.21.15.157:6789/0 conn(0x7f4ee0154060 legacy=0x7f4ee00c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:17.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee0154470 con 0x7f4ee015aa00 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f4ed8001680 con 0x7f4ee015aa00 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 >> v1:172.21.15.136:6789/0 conn(0x7f4ee015aa00 legacy=0x7f4ee0164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 0x7f4ecc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc004810 0x7f4ecc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ecc0051d0 0x7f4ecc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee00c8fd0 con 0x7f4ecc0039c0 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc00c160 con 0x7f4ecc0051d0 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ecc00c360 con 0x7f4ecc004810 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 0x7f4ecc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edf7fe640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc004810 0x7f4ecc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4ee4941640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ecc0051d0 0x7f4ecc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 -- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc004810 msgr2=0x7f4ecc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc004810 0x7f4ecc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 -- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ecc0051d0 msgr2=0x7f4ecc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:17.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ecc0051d0 0x7f4ecc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4edffff640 1 -- 172.21.15.136:0/4120928488 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ecc003530 con 0x7f4ecc0039c0 2024-04-25T11:23:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4ee4941640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ecc0051d0 0x7f4ecc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.821+0000 7f4ee4941640 1 -- 172.21.15.136:0/4120928488 reap_dead start 2024-04-25T11:23:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4edffff640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 0x7f4ecc003e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4ee006db80 tx=0x7f4ed001c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c26ef73efd7376 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:17.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0001aa0 con 0x7f4ecc0039c0 2024-04-25T11:23:17.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed0003b80 con 0x7f4ecc0039c0 2024-04-25T11:23:17.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4edeffd640 1 -- 172.21.15.136:0/4120928488 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0005080 con 0x7f4ecc0039c0 2024-04-25T11:23:17.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 msgr2=0x7f4ecc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:17.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 0x7f4ecc003e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4ee006db80 tx=0x7f4ed001c680 comp rx=0 tx=0).stop 2024-04-25T11:23:17.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- 172.21.15.136:0/4120928488 shutdown_connections 2024-04-25T11:23:17.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/4120928488 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ecc0039c0 0x7f4ecc003e30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- 172.21.15.136:0/4120928488 >> 172.21.15.136:0/4120928488 conn(0x7f4ee00ca5a0 msgr2=0x7f4ee00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:17.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- 172.21.15.136:0/4120928488 shutdown_connections 2024-04-25T11:23:17.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- 172.21.15.136:0/4120928488 wait complete. 2024-04-25T11:23:17.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 Processor -- start 2024-04-25T11:23:17.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.825+0000 7f4ee63a4640 1 -- start start 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc0039c0 0x7f4ee014f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee0152170 0x7f4ee014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee015a0d0 con 0x7f4ee01501e0 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee006df30 con 0x7f4ee0152170 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee0159d80 con 0x7f4ecc0039c0 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38964/0 (socket says 172.21.15.136:38964) 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 -- 172.21.15.136:0/1874943373 learned_addr learned my addr 172.21.15.136:0/1874943373 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edffff640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc0039c0 0x7f4ee014f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee4941640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee0152170 0x7f4ee014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 -- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc0039c0 msgr2=0x7f4ee014f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc0039c0 0x7f4ee014f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 -- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee0152170 msgr2=0x7f4ee014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee0152170 0x7f4ee014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 -- 172.21.15.136:0/1874943373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee014a2b0 con 0x7f4ee01501e0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edf7fe640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80115d0 tx=0x7f4ec8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800f8f0 con 0x7f4ee01501e0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec800fa90 con 0x7f4ee01501e0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee01483c0 con 0x7f4ee01501e0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800fe40 con 0x7f4ee01501e0 2024-04-25T11:23:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.829+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee014ced0 con 0x7f4ee01501e0 2024-04-25T11:23:17.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee006a350 con 0x7f4ee01501e0 2024-04-25T11:23:17.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4ec8015460 con 0x7f4ee01501e0 2024-04-25T11:23:17.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4edcff9640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 0x7f4ec0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:17.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4ec808ff20 con 0x7f4ee01501e0 2024-04-25T11:23:17.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4edffff640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 0x7f4ec0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:17.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.833+0000 7f4edffff640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 0x7f4ec0071940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0005000 tx=0x7f4ed0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:17.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:17.837+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4ec8019030 con 0x7f4ee01501e0 2024-04-25T11:23:18.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.089+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ee0149990 con 0x7f4ee01501e0 2024-04-25T11:23:18.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.089+0000 7f4edcff9640 1 -- 172.21.15.136:0/1874943373 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7f4ec805caa0 con 0x7f4ee01501e0 2024-04-25T11:23:18.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:18.085 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:18.085 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 msgr2=0x7f4ec0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 0x7f4ec0071940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0005000 tx=0x7f4ed0009040 comp rx=0 tx=0).stop 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 msgr2=0x7f4ee0149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80115d0 tx=0x7f4ec8011a90 comp rx=0 tx=0).stop 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 shutdown_connections 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee0152170 0x7f4ee014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:18.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ee01501e0 0x7f4ee0149d70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4ec006f430 0x7f4ec0071940 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 --2- 172.21.15.136:0/1874943373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ecc0039c0 0x7f4ee014f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 >> 172.21.15.136:0/1874943373 conn(0x7f4ee00ca5a0 msgr2=0x7f4ee006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 shutdown_connections 2024-04-25T11:23:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:18.093+0000 7f4ee63a4640 1 -- 172.21.15.136:0/1874943373 wait complete. 2024-04-25T11:23:18.106 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:19.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:23:19.928 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:23:20.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.277+0000 7fe3fc673640 1 Processor -- start 2024-04-25T11:23:20.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.277+0000 7fe3fc673640 1 -- start start 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 0x7fe3f40c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe3f40ca290 0x7fe3f40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f415a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f415d750 con 0x7fe3f40ca290 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f415da30 con 0x7fe3f40b5060 2024-04-25T11:23:20.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fc673640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f415dd10 con 0x7fe3f415a430 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe3f40ca290 0x7fe3f40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe3f40ca290 0x7fe3f40ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38968/0 (socket says 172.21.15.136:38968) 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 -- 172.21.15.136:0/2624624155 learned_addr learned my addr 172.21.15.136:0/2624624155 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fa40f640 1 --2- 172.21.15.136:0/2624624155 >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 0x7fe3f40c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fac10640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 -- 172.21.15.136:0/2624624155 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 msgr2=0x7fe3f415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f415a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 -- 172.21.15.136:0/2624624155 >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 msgr2=0x7fe3f40c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/2624624155 >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 0x7fe3f40c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3f9c0e640 1 -- 172.21.15.136:0/2624624155 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f415dff0 con 0x7fe3f40ca290 2024-04-25T11:23:20.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.281+0000 7fe3fa40f640 1 --2- 172.21.15.136:0/2624624155 >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 0x7fe3f40c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40ca290 0x7fe3f40ca6a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4068ff0 tx=0x7fe3e401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a6edf9a209cd4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:20.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3f940d640 1 -- 172.21.15.136:0/2624624155 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40b5070 con 0x7fe3f40ca290 2024-04-25T11:23:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3f940d640 1 -- 172.21.15.136:0/2624624155 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e40a89d0 con 0x7fe3f40ca290 2024-04-25T11:23:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3f940d640 1 -- 172.21.15.136:0/2624624155 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40a8da0 con 0x7fe3f40ca290 2024-04-25T11:23:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- 172.21.15.136:0/2624624155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40ca290 msgr2=0x7fe3f40ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40ca290 0x7fe3f40ca6a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4068ff0 tx=0x7fe3e401ed10 comp rx=0 tx=0).stop 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- 172.21.15.136:0/2624624155 shutdown_connections 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40ca290 0x7fe3f40ca6a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 --2- 172.21.15.136:0/2624624155 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f415a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 --2- 172.21.15.136:0/2624624155 >> v2:172.21.15.157:3300/0 conn(0x7fe3f40b5060 0x7fe3f40c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- 172.21.15.136:0/2624624155 >> 172.21.15.136:0/2624624155 conn(0x7fe3f4070f70 msgr2=0x7fe3f4068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- 172.21.15.136:0/2624624155 shutdown_connections 2024-04-25T11:23:20.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- 172.21.15.136:0/2624624155 wait complete. 2024-04-25T11:23:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 Processor -- start 2024-04-25T11:23:20.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.285+0000 7fe3fc673640 1 -- start start 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 0x7fe3f40b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f414f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f415ad80 con 0x7fe3f40b5060 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f4159f10 con 0x7fe3f40ca290 2024-04-25T11:23:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f415d3f0 con 0x7fe3f415a430 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38976/0 (socket says 172.21.15.136:38976) 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 -- 172.21.15.136:0/1650193417 learned_addr learned my addr 172.21.15.136:0/1650193417 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 0x7fe3f40b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fac10640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f414f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 -- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 msgr2=0x7fe3f414f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f414f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 -- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 msgr2=0x7fe3f40b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 0x7fe3f40b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 -- 172.21.15.136:0/1650193417 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f41e4190 con 0x7fe3f40b5060 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 0x7fe3f40b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fa40f640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000d800 tx=0x7fe3f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0018070 con 0x7fe3f40b5060 2024-04-25T11:23:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f0004030 con 0x7fe3f40b5060 2024-04-25T11:23:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f414f490 con 0x7fe3f40b5060 2024-04-25T11:23:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0004400 con 0x7fe3f40b5060 2024-04-25T11:23:20.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f41e43a0 con 0x7fe3f40b5060 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f41e3880 con 0x7fe3f40b5060 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.289+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe3f00096a0 con 0x7fe3f40b5060 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.293+0000 7fe3eaffd640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 0x7fe3d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.293+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe3f008f940 con 0x7fe3f40b5060 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.293+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 0x7fe3d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:20.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.293+0000 7fe3f9c0e640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 0x7fe3d8071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40b2000 tx=0x7fe3e40aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:20.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.297+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe3f005c4c0 con 0x7fe3f40b5060 2024-04-25T11:23:20.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.561+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3f406a350 con 0x7fe3f40b5060 2024-04-25T11:23:20.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3eaffd640 1 -- 172.21.15.136:0/1650193417 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7fe3f005bdc0 con 0x7fe3f40b5060 2024-04-25T11:23:20.558 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:20.558 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:20.558 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:20.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 msgr2=0x7fe3d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:20.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 0x7fe3d8071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40b2000 tx=0x7fe3e40aa070 comp rx=0 tx=0).stop 2024-04-25T11:23:20.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 msgr2=0x7fe3f41e3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:20.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000d800 tx=0x7fe3f000dcc0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 shutdown_connections 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe3d806f400 0x7fe3d8071910 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3f415a430 0x7fe3f414f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3f40ca290 0x7fe3f40b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 --2- 172.21.15.136:0/1650193417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3f40b5060 0x7fe3f41e3c50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 >> 172.21.15.136:0/1650193417 conn(0x7fe3f4070f70 msgr2=0x7fe3f40b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 shutdown_connections 2024-04-25T11:23:20.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:20.565+0000 7fe3fc673640 1 -- 172.21.15.136:0/1650193417 wait complete. 2024-04-25T11:23:20.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:22.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:22.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.745+0000 7fd6da8c4640 1 Processor -- start 2024-04-25T11:23:22.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.745+0000 7fd6da8c4640 1 -- start start 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6da8c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6d40ce260 0x7fd6d40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6da8c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d40cbad0 con 0x7fd6d40ca010 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6da8c4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d40c7c20 con 0x7fd6d40c8cd0 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6da8c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d40c8550 con 0x7fd6d40ce260 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d37fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6d40ce260 0x7fd6d40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d8e61640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd6d40ca010 0x7fd6d40ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48264/0 (socket says 172.21.15.136:48264) 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d8e61640 1 -- 172.21.15.136:0/2449029293 learned_addr learned my addr 172.21.15.136:0/2449029293 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:22.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d37fe640 1 -- 172.21.15.136:0/2449029293 >> v1:172.21.15.157:6789/0 conn(0x7fd6d40c8cd0 legacy=0x7fd6d414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:22.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d37fe640 1 -- 172.21.15.136:0/2449029293 >> v1:172.21.15.136:6789/0 conn(0x7fd6d40ca010 legacy=0x7fd6d40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:22.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d37fe640 1 -- 172.21.15.136:0/2449029293 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d40c7f00 con 0x7fd6d40ce260 2024-04-25T11:23:22.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.749+0000 7fd6d2ffd640 1 -- 172.21.15.136:0/2449029293 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453091404 0 0) 0x7fd6d40cbad0 con 0x7fd6d40ca010 2024-04-25T11:23:22.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6d37fe640 1 --2- 172.21.15.136:0/2449029293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ce260 0x7fd6d40ce670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80097c0 tx=0x7fd6c801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6734ad4c49e9ae13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:22.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6d2ffd640 1 -- 172.21.15.136:0/2449029293 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80b5070 con 0x7fd6d40ce260 2024-04-25T11:23:22.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6d2ffd640 1 -- 172.21.15.136:0/2449029293 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c80a8990 con 0x7fd6d40ce260 2024-04-25T11:23:22.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6d2ffd640 1 -- 172.21.15.136:0/2449029293 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80a8d60 con 0x7fd6d40ce260 2024-04-25T11:23:22.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- 172.21.15.136:0/2449029293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ce260 msgr2=0x7fd6d40ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:22.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/2449029293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ce260 0x7fd6d40ce670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80097c0 tx=0x7fd6c801ec90 comp rx=0 tx=0).stop 2024-04-25T11:23:22.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- 172.21.15.136:0/2449029293 shutdown_connections 2024-04-25T11:23:22.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/2449029293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ce260 0x7fd6d40ce670 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:22.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- 172.21.15.136:0/2449029293 >> 172.21.15.136:0/2449029293 conn(0x7fd6d406f9d0 msgr2=0x7fd6d406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:22.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- 172.21.15.136:0/2449029293 shutdown_connections 2024-04-25T11:23:22.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- 172.21.15.136:0/2449029293 wait complete. 2024-04-25T11:23:22.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 Processor -- start 2024-04-25T11:23:22.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.753+0000 7fd6da8c4640 1 -- start start 2024-04-25T11:23:22.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 0x7fd6d41f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:22.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:22.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d40ce260 0x7fd6d4148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:22.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d40cb720 con 0x7fd6d40ca010 2024-04-25T11:23:22.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d40c91e0 con 0x7fd6d40c8cd0 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d414d1d0 con 0x7fd6d40ce260 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38990/0 (socket says 172.21.15.136:38990) 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 -- 172.21.15.136:0/1531375267 learned_addr learned my addr 172.21.15.136:0/1531375267 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d8e61640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d40ce260 0x7fd6d4148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d3fff640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 0x7fd6d41f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 -- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d40ce260 msgr2=0x7fd6d4148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d40ce260 0x7fd6d4148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 -- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 msgr2=0x7fd6d41f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 0x7fd6d41f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 -- 172.21.15.136:0/1531375267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41f2e50 con 0x7fd6d40ca010 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d3fff640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 0x7fd6d41f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d37fe640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801b8a0 tx=0x7fd6c801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:22.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80bf070 con 0x7fd6d40ca010 2024-04-25T11:23:22.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c80b0820 con 0x7fd6d40ca010 2024-04-25T11:23:22.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d414cea0 con 0x7fd6d40ca010 2024-04-25T11:23:22.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80b5040 con 0x7fd6d40ca010 2024-04-25T11:23:22.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d414d700 con 0x7fd6d40ca010 2024-04-25T11:23:22.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.757+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d406a350 con 0x7fd6d40ca010 2024-04-25T11:23:22.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.761+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6c80b0c10 con 0x7fd6d40ca010 2024-04-25T11:23:22.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.761+0000 7fd6d0ff9640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 0x7fd6c0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:22.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.761+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd6c812be70 con 0x7fd6d40ca010 2024-04-25T11:23:22.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.761+0000 7fd6d3fff640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 0x7fd6c0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:22.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.761+0000 7fd6d3fff640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 0x7fd6c0071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000f70 tx=0x7fd6c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:22.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:22.765+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6c80187f0 con 0x7fd6d40ca010 2024-04-25T11:23:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.009+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d4156790 con 0x7fd6d40ca010 2024-04-25T11:23:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.009+0000 7fd6d0ff9640 1 -- 172.21.15.136:0/1531375267 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6921 (secure 0 0 0) 0x7fd6c80f8970 con 0x7fd6d40ca010 2024-04-25T11:23:23.004 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:23.004 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:21:12.989866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":4,"incarnation":18,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6835/430807169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":430807169},{"type":"v1","addr":"172.21.15.136:6835","nonce":430807169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4257,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 19 2024-04-25T11:23:23.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 msgr2=0x7fd6c0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 0x7fd6c0071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000f70 tx=0x7fd6c4009040 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 msgr2=0x7fd6d406c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801b8a0 tx=0x7fd6c801ba30 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 shutdown_connections 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6c006f430 0x7fd6c0071940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d40ce260 0x7fd6d4148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d40ca010 0x7fd6d406c840 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 --2- 172.21.15.136:0/1531375267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d40c8cd0 0x7fd6d41f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 >> 172.21.15.136:0/1531375267 conn(0x7fd6d406f9d0 msgr2=0x7fd6d40cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 shutdown_connections 2024-04-25T11:23:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:23.013+0000 7fd6da8c4640 1 -- 172.21.15.136:0/1531375267 wait complete. 2024-04-25T11:23:23.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:23:25.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 Processor -- start 2024-04-25T11:23:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 -- start start 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3bec0b5060 0x7f3bec0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3bec15ab40 0x7f3bec0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bec15d750 con 0x7f3bec0ca290 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3bec15da30 con 0x7f3bec0b5060 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf3874640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3bec15dd10 con 0x7f3bec15ab40 2024-04-25T11:23:25.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3bec0b5060 0x7f3bec0c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3bec0b5060 0x7f3bec0c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49494/0 (socket says 172.21.15.136:49494) 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 -- 172.21.15.136:0/792344534 learned_addr learned my addr 172.21.15.136:0/792344534 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1e11640 1 --2- 172.21.15.136:0/792344534 >> v2:172.21.15.157:3300/0 conn(0x7f3bec15ab40 0x7f3bec0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 -- 172.21.15.136:0/792344534 >> v2:172.21.15.157:3300/0 conn(0x7f3bec15ab40 msgr2=0x7f3bec0ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 --2- 172.21.15.136:0/792344534 >> v2:172.21.15.157:3300/0 conn(0x7f3bec15ab40 0x7f3bec0ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 -- 172.21.15.136:0/792344534 >> v1:172.21.15.157:6789/0 conn(0x7f3bec0ca290 legacy=0x7f3bec15a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.217+0000 7f3bf1610640 1 -- 172.21.15.136:0/792344534 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bec15dff0 con 0x7f3bec0b5060 2024-04-25T11:23:25.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf1610640 1 --2- 172.21.15.136:0/792344534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec0b5060 0x7f3bec0c4520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0b4fe0 tx=0x7f3be801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19089abb23ab0808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3be3fff640 1 -- 172.21.15.136:0/792344534 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80b5070 con 0x7f3bec0b5060 2024-04-25T11:23:25.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3be3fff640 1 -- 172.21.15.136:0/792344534 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be80a89d0 con 0x7f3bec0b5060 2024-04-25T11:23:25.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3be3fff640 1 -- 172.21.15.136:0/792344534 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80a8da0 con 0x7f3bec0b5060 2024-04-25T11:23:25.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 -- 172.21.15.136:0/792344534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec0b5060 msgr2=0x7f3bec0c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 --2- 172.21.15.136:0/792344534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec0b5060 0x7f3bec0c4520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0b4fe0 tx=0x7f3be801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 -- 172.21.15.136:0/792344534 shutdown_connections 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 --2- 172.21.15.136:0/792344534 >> v2:172.21.15.157:3300/0 conn(0x7f3bec15ab40 0x7f3bec0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 --2- 172.21.15.136:0/792344534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec0b5060 0x7f3bec0c4520 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 -- 172.21.15.136:0/792344534 >> 172.21.15.136:0/792344534 conn(0x7f3bec070f70 msgr2=0x7f3bec068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 -- 172.21.15.136:0/792344534 shutdown_connections 2024-04-25T11:23:25.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.221+0000 7f3bf3874640 1 -- 172.21.15.136:0/792344534 wait complete. 2024-04-25T11:23:25.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 Processor -- start 2024-04-25T11:23:25.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 -- start start 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bec0b5060 0x7f3bec1e32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bec0ca290 0x7f3bec1524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec15d3f0 con 0x7f3bec15ab40 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec159f10 con 0x7f3bec0ca290 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf3874640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3bec0c4a60 con 0x7f3bec0b5060 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49496/0 (socket says 172.21.15.136:49496) 2024-04-25T11:23:25.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 -- 172.21.15.136:0/219248559 learned_addr learned my addr 172.21.15.136:0/219248559 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1610640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bec0b5060 0x7f3bec1e32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf0e0f640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bec0ca290 0x7f3bec1524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 -- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bec0b5060 msgr2=0x7f3bec1e32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bec0b5060 0x7f3bec1e32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 -- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bec0ca290 msgr2=0x7f3bec1524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bec0ca290 0x7f3bec1524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 -- 172.21.15.136:0/219248559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bec1535f0 con 0x7f3bec15ab40 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3bf1e11640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3be4007d40 tx=0x7f3be400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.225+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4014420 con 0x7f3bec15ab40 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be4019070 con 0x7f3bec15ab40 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bec153000 con 0x7f3bec15ab40 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4014820 con 0x7f3bec15ab40 2024-04-25T11:23:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bec1e39c0 con 0x7f3bec15ab40 2024-04-25T11:23:25.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bec1e37f0 con 0x7f3bec15ab40 2024-04-25T11:23:25.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3be40149c0 con 0x7f3bec15ab40 2024-04-25T11:23:25.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3be1ffb640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 0x7f3bd0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f3be408ff50 con 0x7f3bec15ab40 2024-04-25T11:23:25.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.229+0000 7f3bf1610640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 0x7f3bd0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.233+0000 7f3bf1610640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 0x7f3bd0071910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3be80b2000 tx=0x7f3be8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.233+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3be405c9e0 con 0x7f3bec15ab40 2024-04-25T11:23:25.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.457+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bec1531d0 con 0x7f3bec15ab40 2024-04-25T11:23:25.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.457+0000 7f3be1ffb640 1 -- 172.21.15.136:0/219248559 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6093 (secure 0 0 0) 0x7f3be405c2e0 con 0x7f3bec15ab40 2024-04-25T11:23:25.454 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:25.454 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:23.788026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:25.454 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 20 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 msgr2=0x7f3bd0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 0x7f3bd0071910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3be80b2000 tx=0x7f3be8008040 comp rx=0 tx=0).stop 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 msgr2=0x7f3bec1e3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3be4007d40 tx=0x7f3be400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 shutdown_connections 2024-04-25T11:23:25.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bec15ab40 0x7f3bec1e3c80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bec0ca290 0x7f3bec1524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bd006f400 0x7f3bd0071910 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 --2- 172.21.15.136:0/219248559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bec0b5060 0x7f3bec1e32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 >> 172.21.15.136:0/219248559 conn(0x7f3bec070f70 msgr2=0x7f3bec15a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 shutdown_connections 2024-04-25T11:23:25.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.461+0000 7f3bf3874640 1 -- 172.21.15.136:0/219248559 wait complete. 2024-04-25T11:23:25.474 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-25T11:23:25.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:23:25.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:25.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 Processor -- start 2024-04-25T11:23:25.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 -- start start 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f48080c9b50 0x7f480815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f480815c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080c8a20 con 0x7f48080ca010 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f480806c670 con 0x7f48080c9b50 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480f98c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f48080c8d00 con 0x7f480806d210 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480d728640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f48080ca010 0x7f480806cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49666/0 (socket says 172.21.15.136:49666) 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480d728640 1 -- 172.21.15.136:0/3394868294 learned_addr learned my addr 172.21.15.136:0/3394868294 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:25.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480cf27640 1 --2- 172.21.15.136:0/3394868294 >> v2:172.21.15.157:3300/0 conn(0x7f48080c9b50 0x7f480815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f480df29640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f480815c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324165331 0 0) 0x7f48080c8a20 con 0x7f48080ca010 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.633+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47f4003530 con 0x7f48080ca010 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2063182909 0 0) 0x7f47f4003530 con 0x7f48080ca010 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 msgr2=0x7f480815c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f480815c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 >> v2:172.21.15.157:3300/0 conn(0x7f48080c9b50 msgr2=0x7f480815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 --2- 172.21.15.136:0/3394868294 >> v2:172.21.15.157:3300/0 conn(0x7f48080c9b50 0x7f480815bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4808074d10 con 0x7f48080ca010 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f48040016a0 con 0x7f48080ca010 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 >> v1:172.21.15.136:6789/0 conn(0x7f48080ca010 legacy=0x7f480806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f40039c0 0x7f47f4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f47f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47f4004d50 0x7f47f4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f480806c670 con 0x7f47f4004d50 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f48080c8d00 con 0x7f47f40039c0 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47f400ba00 con 0x7f480806d210 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47f4004d50 0x7f47f4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480cf27640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f47f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480d728640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f40039c0 0x7f47f4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 -- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 msgr2=0x7f47f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480806d210 0x7f47f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 -- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f40039c0 msgr2=0x7f47f4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47f40039c0 0x7f47f4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 -- 172.21.15.136:0/3394868294 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f4003530 con 0x7f47f4004d50 2024-04-25T11:23:25.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f480df29640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47f4004d50 0x7f47f4005160 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f480806e920 tx=0x7f480001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7b840d5c4a8500 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000b2070 con 0x7f47f4004d50 2024-04-25T11:23:25.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.637+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48000a5c10 con 0x7f47f4004d50 2024-04-25T11:23:25.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f47fffff640 1 -- 172.21.15.136:0/3394868294 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800009c10 con 0x7f47f4004d50 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- 172.21.15.136:0/3394868294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47f4004d50 msgr2=0x7f47f4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 --2- 172.21.15.136:0/3394868294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47f4004d50 0x7f47f4005160 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f480806e920 tx=0x7f480001cba0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480df29640 1 -- 172.21.15.136:0/3394868294 reap_dead start 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- 172.21.15.136:0/3394868294 shutdown_connections 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- 172.21.15.136:0/3394868294 >> 172.21.15.136:0/3394868294 conn(0x7f48080ca5a0 msgr2=0x7f48080ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:25.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- 172.21.15.136:0/3394868294 shutdown_connections 2024-04-25T11:23:25.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- 172.21.15.136:0/3394868294 wait complete. 2024-04-25T11:23:25.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 Processor -- start 2024-04-25T11:23:25.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.641+0000 7f480f98c640 1 -- start start 2024-04-25T11:23:25.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 0x7f48081e7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 0x7f48081e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f480815c520 con 0x7f4808156be0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f48080c9240 con 0x7f48080cc4c0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f480815c6a0 con 0x7f48080cc8d0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49516/0 (socket says 172.21.15.136:49516) 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 -- 172.21.15.136:0/3883370994 learned_addr learned my addr 172.21.15.136:0/3883370994 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480cf27640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 0x7f48081e7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480df29640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 0x7f48081e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 -- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 msgr2=0x7f48081e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 0x7f48081e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 -- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 msgr2=0x7f48081e7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 0x7f48081e7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 -- 172.21.15.136:0/3883370994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480815d300 con 0x7f4808156be0 2024-04-25T11:23:25.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480cf27640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 0x7f48081e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480df29640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 0x7f48081e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480d728640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4804001230 tx=0x7f480400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480400c8d0 con 0x7f4808156be0 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4804018070 con 0x7f4808156be0 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480815d5a0 con 0x7f4808156be0 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480400cc90 con 0x7f4808156be0 2024-04-25T11:23:25.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f480815db60 con 0x7f4808156be0 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f480806a350 con 0x7f4808156be0 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.645+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f480401d460 con 0x7f4808156be0 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.649+0000 7f47fdffb640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 0x7f47ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.649+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f480408f680 con 0x7f4808156be0 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.649+0000 7f480cf27640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 0x7f47ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.649+0000 7f480cf27640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 0x7f47ec071910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f480814e610 tx=0x7f47f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.653+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f480405c110 con 0x7f4808156be0 2024-04-25T11:23:25.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48080c94e0 con 0x7f4808156be0 2024-04-25T11:23:25.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f47fdffb640 1 -- 172.21.15.136:0/3883370994 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6093 (secure 0 0 0) 0x7f4804064190 con 0x7f4808156be0 2024-04-25T11:23:25.871 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:25.871 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:23.788026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:25.872 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 20 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 msgr2=0x7f47ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 0x7f47ec071910 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f480814e610 tx=0x7f47f8008040 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 msgr2=0x7f48081f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4804001230 tx=0x7f480400ef80 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 shutdown_connections 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47ec06f400 0x7f47ec071910 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f48080cc8d0 0x7f48081e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48080cc4c0 0x7f48081e7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 --2- 172.21.15.136:0/3883370994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4808156be0 0x7f48081f29e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:25.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 >> 172.21.15.136:0/3883370994 conn(0x7f48080ca5a0 msgr2=0x7f4808075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:25.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 shutdown_connections 2024-04-25T11:23:25.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:25.877+0000 7f480f98c640 1 -- 172.21.15.136:0/3883370994 wait complete. 2024-04-25T11:23:25.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4254": { "addr": "172.21.15.136:6833/3353376405", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 3353376405, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 3353376405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4254, "incarnation": 20, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4260": { "addr": "172.21.15.136:6837/1368493675", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1368493675, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1368493675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:23:23.788026+0000", "qdb_cluster": [ 4260, 4269, 4297, 4303 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 4260, "mds_4": 4254 } } } ], "standbys": [] } 2024-04-25T11:23:27.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:28.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 Processor -- start 2024-04-25T11:23:28.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 -- start start 2024-04-25T11:23:28.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 0x7f73740ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73740cbad0 con 0x7f73740c8cd0 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f73740c8550 con 0x7f73740ce260 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737baab640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73740c7c20 con 0x7f73740ca010 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f7379046640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f7379046640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37634/0 (socket says 172.21.15.136:37634) 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f7379046640 1 -- 172.21.15.136:0/186161465 learned_addr learned my addr 172.21.15.136:0/186161465 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:28.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737a048640 1 --2- 172.21.15.136:0/186161465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 0x7f73740ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.073+0000 7f737a048640 1 -- 172.21.15.136:0/186161465 >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 msgr2=0x7f73740ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:28.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f737a048640 1 --2- 172.21.15.136:0/186161465 >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f737a048640 1 -- 172.21.15.136:0/186161465 >> v1:172.21.15.157:6789/0 conn(0x7f73740c8cd0 legacy=0x7f737414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f737a048640 1 -- 172.21.15.136:0/186161465 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73740c7f00 con 0x7f73740ca010 2024-04-25T11:23:28.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f7379046640 1 --2- 172.21.15.136:0/186161465 >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:23:28.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f737a048640 1 --2- 172.21.15.136:0/186161465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 0x7f73740ca420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73740c86d0 tx=0x7f736c01f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7d5e358d73f20c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f7378845640 1 -- 172.21.15.136:0/186161465 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c0b4070 con 0x7f73740ca010 2024-04-25T11:23:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f7378845640 1 -- 172.21.15.136:0/186161465 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736c0af600 con 0x7f73740ca010 2024-04-25T11:23:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.077+0000 7f7378845640 1 -- 172.21.15.136:0/186161465 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c0af9d0 con 0x7f73740ca010 2024-04-25T11:23:28.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- 172.21.15.136:0/186161465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 msgr2=0x7f73740ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 --2- 172.21.15.136:0/186161465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 0x7f73740ca420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73740c86d0 tx=0x7f736c01f130 comp rx=0 tx=0).stop 2024-04-25T11:23:28.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- 172.21.15.136:0/186161465 shutdown_connections 2024-04-25T11:23:28.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 --2- 172.21.15.136:0/186161465 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ca010 0x7f73740ca420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 --2- 172.21.15.136:0/186161465 >> v2:172.21.15.157:3300/0 conn(0x7f73740ce260 0x7f73740ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- 172.21.15.136:0/186161465 >> 172.21.15.136:0/186161465 conn(0x7f737406f9d0 msgr2=0x7f737406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:28.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- 172.21.15.136:0/186161465 shutdown_connections 2024-04-25T11:23:28.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- 172.21.15.136:0/186161465 wait complete. 2024-04-25T11:23:28.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 Processor -- start 2024-04-25T11:23:28.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.081+0000 7f737baab640 1 -- start start 2024-04-25T11:23:28.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 0x7f7374155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ce260 0x7f737414a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f73740cb720 con 0x7f73740ca010 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f73740ca960 con 0x7f73740c8cd0 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7374155d10 con 0x7f73740ce260 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49530/0 (socket says 172.21.15.136:49530) 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 -- 172.21.15.136:0/4216794151 learned_addr learned my addr 172.21.15.136:0/4216794151 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737a048640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ce260 0x7f737414a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379847640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 0x7f7374155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 -- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ce260 msgr2=0x7f737414a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ce260 0x7f737414a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 -- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 msgr2=0x7f7374155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 0x7f7374155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 -- 172.21.15.136:0/4216794151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7374155840 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379847640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 0x7f7374155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f7379046640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f736400e980 tx=0x7f736400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364017070 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736400ca00 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736400cdf0 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73740cd050 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7374154b00 con 0x7f73740ca010 2024-04-25T11:23:28.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737406a350 con 0x7f73740ca010 2024-04-25T11:23:28.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.085+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f736401a070 con 0x7f73740ca010 2024-04-25T11:23:28.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.089+0000 7f736a7fc640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 0x7f7358071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:28.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.089+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f736408f8e0 con 0x7f73740ca010 2024-04-25T11:23:28.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.089+0000 7f7379847640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 0x7f7358071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:28.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.089+0000 7f7379847640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 0x7f7358071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7370000f70 tx=0x7f7370009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:28.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.093+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f736405c400 con 0x7f73740ca010 2024-04-25T11:23:28.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.337+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73741645e0 con 0x7f73740ca010 2024-04-25T11:23:28.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f736a7fc640 1 -- 172.21.15.136:0/4216794151 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6093 (secure 0 0 0) 0x7f736405bd00 con 0x7f73740ca010 2024-04-25T11:23:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:28.334 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:23.788026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:28.335 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 20 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 msgr2=0x7f7358071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 0x7f7358071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7370000f70 tx=0x7f7370009040 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 msgr2=0x7f737406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f736400e980 tx=0x7f736400ee40 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 shutdown_connections 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735806f430 0x7f7358071940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f73740ce260 0x7f737414a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73740ca010 0x7f737406d9c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 --2- 172.21.15.136:0/4216794151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73740c8cd0 0x7f7374155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 >> 172.21.15.136:0/4216794151 conn(0x7f737406f9d0 msgr2=0x7f73740cc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 shutdown_connections 2024-04-25T11:23:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:28.341+0000 7f737baab640 1 -- 172.21.15.136:0/4216794151 wait complete. 2024-04-25T11:23:28.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:30.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:30.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.545+0000 7fae29eeb640 1 Processor -- start 2024-04-25T11:23:30.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.545+0000 7fae29eeb640 1 -- start start 2024-04-25T11:23:30.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae29eeb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fae240ce260 0x7fae240ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae29eeb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae240cbad0 con 0x7fae240ca010 2024-04-25T11:23:30.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae29eeb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae240c7c20 con 0x7fae240c8cd0 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae29eeb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fae240c8550 con 0x7fae240ce260 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fae240ca010 0x7fae240ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:49680/0 (socket says 172.21.15.136:49680) 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 -- 172.21.15.136:0/3679504356 learned_addr learned my addr 172.21.15.136:0/3679504356 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae22ffd640 1 --2- 172.21.15.136:0/3679504356 >> v2:172.21.15.157:3300/0 conn(0x7fae240ce260 0x7fae240ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875748842 0 0) 0x7fae240cbad0 con 0x7fae240ca010 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae0c003530 con 0x7fae240ca010 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 691022462 0 0) 0x7fae0c003530 con 0x7fae240ca010 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 >> v2:172.21.15.157:3300/0 conn(0x7fae240ce260 msgr2=0x7fae240ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 --2- 172.21.15.136:0/3679504356 >> v2:172.21.15.157:3300/0 conn(0x7fae240ce260 0x7fae240ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 >> v1:172.21.15.157:6789/0 conn(0x7fae240c8cd0 legacy=0x7fae2414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae240c7f00 con 0x7fae240ca010 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fae1c001a00 con 0x7fae240ca010 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 >> v1:172.21.15.136:6789/0 conn(0x7fae240ca010 legacy=0x7fae240ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae0c0039c0 0x7fae0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae0c004810 0x7fae0c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae0c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fae240c8550 con 0x7fae0c0051d0 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c0099f0 con 0x7fae0c004810 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fae0c009bf0 con 0x7fae0c0039c0 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae0c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae22ffd640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae0c004810 0x7fae0c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae237fe640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae0c0039c0 0x7fae0c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 -- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae0c0039c0 msgr2=0x7fae0c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae0c0039c0 0x7fae0c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 -- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae0c004810 msgr2=0x7fae0c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae0c004810 0x7fae0c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae23fff640 1 -- 172.21.15.136:0/3679504356 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c003530 con 0x7fae0c0051d0 2024-04-25T11:23:30.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.549+0000 7fae237fe640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae0c0039c0 0x7fae0c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae23fff640 1 -- 172.21.15.136:0/3679504356 reap_dead start 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae23fff640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae0c00dbe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00a840 tx=0x7fae1c01d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2720db3c9fb0006c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00a460 con 0x7fae0c0051d0 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1c0ac640 con 0x7fae0c0051d0 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae227fc640 1 -- 172.21.15.136:0/3679504356 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0aca30 con 0x7fae0c0051d0 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 -- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 msgr2=0x7fae0c00dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae0c00dbe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00a840 tx=0x7fae1c01d2a0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 -- 172.21.15.136:0/3679504356 shutdown_connections 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 --2- 172.21.15.136:0/3679504356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae0c00dbe0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 -- 172.21.15.136:0/3679504356 >> 172.21.15.136:0/3679504356 conn(0x7fae2406f9d0 msgr2=0x7fae2406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:30.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 -- 172.21.15.136:0/3679504356 shutdown_connections 2024-04-25T11:23:30.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.553+0000 7fae29eeb640 1 -- 172.21.15.136:0/3679504356 wait complete. 2024-04-25T11:23:30.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 Processor -- start 2024-04-25T11:23:30.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- start start 2024-04-25T11:23:30.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae241fa300 0x7fae24148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae240cc0d0 0x7fae240cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fae240c91e0 con 0x7fae0c0051d0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fae240cbcd0 con 0x7fae240cc0d0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fae240c9360 con 0x7fae241fa300 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49548/0 (socket says 172.21.15.136:49548) 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 -- 172.21.15.136:0/1560422487 learned_addr learned my addr 172.21.15.136:0/1560422487 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae22ffd640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae241fa300 0x7fae24148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae23fff640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae240cc0d0 0x7fae240cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 -- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae241fa300 msgr2=0x7fae24148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae241fa300 0x7fae24148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 -- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae240cc0d0 msgr2=0x7fae240cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae240cc0d0 0x7fae240cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 -- 172.21.15.136:0/1560422487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae240cdd00 con 0x7fae0c0051d0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae237fe640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fae1400d540 tx=0x7fae1400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:30.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14003620 con 0x7fae0c0051d0 2024-04-25T11:23:30.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae14017070 con 0x7fae0c0051d0 2024-04-25T11:23:30.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae241f9820 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae140039e0 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.557+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae240cdfe0 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fae14003c00 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae0bfff640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 0x7fae100718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2406a350 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fae1408eda0 con 0x7fae0c0051d0 2024-04-25T11:23:30.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae22ffd640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 0x7fae100718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:30.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.561+0000 7fae22ffd640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 0x7fae100718b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fae24158590 tx=0x7fae18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:30.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.565+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fae1405b8f0 con 0x7fae0c0051d0 2024-04-25T11:23:30.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.809+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae2414cf20 con 0x7fae0c0051d0 2024-04-25T11:23:30.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.809+0000 7fae0bfff640 1 -- 172.21.15.136:0/1560422487 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6095 (secure 0 0 0) 0x7fae1405b1f0 con 0x7fae0c0051d0 2024-04-25T11:23:30.806 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:30.806 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:28.800372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:resolve","state_seq":51,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:23:30.806 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 21 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 msgr2=0x7fae100718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 0x7fae100718b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fae24158590 tx=0x7fae18008040 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 msgr2=0x7fae2406d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fae1400d540 tx=0x7fae1400da00 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 shutdown_connections 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae240cc0d0 0x7fae240cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fae1006f3a0 0x7fae100718b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fae241fa300 0x7fae24148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 --2- 172.21.15.136:0/1560422487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c0051d0 0x7fae2406d910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:30.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 >> 172.21.15.136:0/1560422487 conn(0x7fae2406f9d0 msgr2=0x7fae24157010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 shutdown_connections 2024-04-25T11:23:30.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:30.813+0000 7fae29eeb640 1 -- 172.21.15.136:0/1560422487 wait complete. 2024-04-25T11:23:30.828 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:32.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:32.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 Processor -- start 2024-04-25T11:23:32.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 -- start start 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f47500c96e0 0x7f47501540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47500c9af0 con 0x7f47500c79e0 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f475006cc10 con 0x7f47500c96e0 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f4755919640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47500cb220 con 0x7f475015aae0 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474effd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f47500c96e0 0x7f47501540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56326/0 (socket says 172.21.15.136:56326) 2024-04-25T11:23:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474f7fe640 1 -- 172.21.15.136:0/1423838983 learned_addr learned my addr 172.21.15.136:0/1423838983 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474effd640 1 -- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 msgr2=0x7f47500c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474effd640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474effd640 1 -- 172.21.15.136:0/1423838983 >> v1:172.21.15.157:6789/0 conn(0x7f47500c79e0 legacy=0x7f475015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474effd640 1 -- 172.21.15.136:0/1423838983 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47500cb500 con 0x7f47500c96e0 2024-04-25T11:23:33.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.004+0000 7f474f7fe640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:23:33.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f474effd640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f47501540d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f47400097c0 tx=0x7f474001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beafef4a26511bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:33.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f474dffb640 1 -- 172.21.15.136:0/1423838983 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400b5070 con 0x7f47500c96e0 2024-04-25T11:23:33.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f474dffb640 1 -- 172.21.15.136:0/1423838983 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47400a8990 con 0x7f47500c96e0 2024-04-25T11:23:33.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f474dffb640 1 -- 172.21.15.136:0/1423838983 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400a8d60 con 0x7f47500c96e0 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 -- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 msgr2=0x7f47501540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f47501540d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f47400097c0 tx=0x7f474001ec90 comp rx=0 tx=0).stop 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 -- 172.21.15.136:0/1423838983 shutdown_connections 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f475015aae0 0x7f47500c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 --2- 172.21.15.136:0/1423838983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f47501540d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 -- 172.21.15.136:0/1423838983 >> 172.21.15.136:0/1423838983 conn(0x7f475006f9d0 msgr2=0x7f475006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 -- 172.21.15.136:0/1423838983 shutdown_connections 2024-04-25T11:23:33.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.008+0000 7f4755919640 1 -- 172.21.15.136:0/1423838983 wait complete. 2024-04-25T11:23:33.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 Processor -- start 2024-04-25T11:23:33.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- start start 2024-04-25T11:23:33.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47500c79e0 0x7f47500c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f4750158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f47500cf0f0 con 0x7f475015aae0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4750154610 con 0x7f47500c96e0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47500ced40 con 0x7f47500c79e0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49564/0 (socket says 172.21.15.136:49564) 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 -- 172.21.15.136:0/2082912060 learned_addr learned my addr 172.21.15.136:0/2082912060 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474effd640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47500c79e0 0x7f47500c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474e7fc640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f4750158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 -- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47500c79e0 msgr2=0x7f47500c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47500c79e0 0x7f47500c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 -- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 msgr2=0x7f4750158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f4750158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 -- 172.21.15.136:0/2082912060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47500cd1c0 con 0x7f475015aae0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f474f7fe640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f474800efc0 tx=0x7f474800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474800cbe0 con 0x7f475015aae0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f474800cd80 con 0x7f475015aae0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47500c7480 con 0x7f475015aae0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748003be0 con 0x7f475015aae0 2024-04-25T11:23:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.012+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47500c7760 con 0x7f475015aae0 2024-04-25T11:23:33.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475006a350 con 0x7f475015aae0 2024-04-25T11:23:33.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f474801b030 con 0x7f475015aae0 2024-04-25T11:23:33.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f4737fff640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 0x7f473c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:33.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f474808f9c0 con 0x7f475015aae0 2024-04-25T11:23:33.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f474effd640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 0x7f473c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:33.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.016+0000 7f474effd640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 0x7f473c071910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47400b2000 tx=0x7f4740008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:33.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.020+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4748091e20 con 0x7f475015aae0 2024-04-25T11:23:33.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.288+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47501579a0 con 0x7f475015aae0 2024-04-25T11:23:33.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.288+0000 7f4737fff640 1 -- 172.21.15.136:0/2082912060 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6095 (secure 0 0 0) 0x7f4748022020 con 0x7f475015aae0 2024-04-25T11:23:33.284 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:33.285 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:28.800372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:resolve","state_seq":51,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:23:33.285 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 21 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 msgr2=0x7f473c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 0x7f473c071910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47400b2000 tx=0x7f4740008040 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 msgr2=0x7f4750142950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f474800efc0 tx=0x7f474800c490 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 shutdown_connections 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475015aae0 0x7f4750142950 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47500c96e0 0x7f4750158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f400 0x7f473c071910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 --2- 172.21.15.136:0/2082912060 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47500c79e0 0x7f47500c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 >> 172.21.15.136:0/2082912060 conn(0x7f475006f9d0 msgr2=0x7f4750150790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:33.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 shutdown_connections 2024-04-25T11:23:33.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:33.292+0000 7f4755919640 1 -- 172.21.15.136:0/2082912060 wait complete. 2024-04-25T11:23:33.306 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:35.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:35.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 Processor -- start 2024-04-25T11:23:35.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 -- start start 2024-04-25T11:23:35.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f86900b5060 0x7f86900c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f869015d750 con 0x7f86900ca290 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f869015da30 con 0x7f869015ab40 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8697773640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f869015dd10 con 0x7f86900b5060 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8694d0e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f86900ca290 0x7f869015a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59106/0 (socket says 172.21.15.136:59106) 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f8694d0e640 1 -- 172.21.15.136:0/3277170663 learned_addr learned my addr 172.21.15.136:0/3277170663 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.476+0000 7f869550f640 1 --2- 172.21.15.136:0/3277170663 >> v2:172.21.15.157:3300/0 conn(0x7f86900b5060 0x7f86900c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465586846 0 0) 0x7f869015d750 con 0x7f86900ca290 2024-04-25T11:23:35.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f867c003530 con 0x7f86900ca290 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3976381790 0 0) 0x7f867c003530 con 0x7f86900ca290 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 >> v2:172.21.15.157:3300/0 conn(0x7f86900b5060 msgr2=0x7f86900c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 --2- 172.21.15.136:0/3277170663 >> v2:172.21.15.157:3300/0 conn(0x7f86900b5060 0x7f86900c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 >> v1:172.21.15.157:6789/0 conn(0x7f869015ab40 legacy=0x7f8690164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869015dff0 con 0x7f86900ca290 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8680001680 con 0x7f86900ca290 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 >> v1:172.21.15.136:6789/0 conn(0x7f86900ca290 legacy=0x7f869015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f867c003a10 0x7f867c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c004810 0x7f867c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 0x7f867c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f869015dd10 con 0x7f867c005280 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f867c00c190 con 0x7f867c004810 2024-04-25T11:23:35.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f867c00c390 con 0x7f867c003a10 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 0x7f867c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f869550f640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f867c003a10 0x7f867c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8694d0e640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c004810 0x7f867c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 -- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f867c003a10 msgr2=0x7f867c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f867c003a10 0x7f867c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 -- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c004810 msgr2=0x7f867c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c004810 0x7f867c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 -- 172.21.15.136:0/3277170663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c003530 con 0x7f867c005280 2024-04-25T11:23:35.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 -- 172.21.15.136:0/3277170663 reap_dead start 2024-04-25T11:23:35.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8695d10640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 0x7f867c00b9d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8690068fc0 tx=0x7f868801c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382551c953a47da3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:35.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688001a90 con 0x7f867c005280 2024-04-25T11:23:35.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8688003b80 con 0x7f867c005280 2024-04-25T11:23:35.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.480+0000 7f8687fff640 1 -- 172.21.15.136:0/3277170663 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688005020 con 0x7f867c005280 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 msgr2=0x7f867c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 0x7f867c00b9d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8690068fc0 tx=0x7f868801c720 comp rx=0 tx=0).stop 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- 172.21.15.136:0/3277170663 shutdown_connections 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 --2- 172.21.15.136:0/3277170663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c005280 0x7f867c00b9d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- 172.21.15.136:0/3277170663 >> 172.21.15.136:0/3277170663 conn(0x7f8690070f70 msgr2=0x7f8690068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- 172.21.15.136:0/3277170663 shutdown_connections 2024-04-25T11:23:35.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- 172.21.15.136:0/3277170663 wait complete. 2024-04-25T11:23:35.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 Processor -- start 2024-04-25T11:23:35.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.484+0000 7f8697773640 1 -- start start 2024-04-25T11:23:35.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c005280 0x7f86901f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8690150b80 0x7f86900cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f869015d3f0 con 0x7f869014c570 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8690159f10 con 0x7f867c005280 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f869015d570 con 0x7f8690150b80 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46036/0 (socket says 172.21.15.136:46036) 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 -- 172.21.15.136:0/794882108 learned_addr learned my addr 172.21.15.136:0/794882108 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8694d0e640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8690150b80 0x7f86900cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f869550f640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c005280 0x7f86901f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 -- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8690150b80 msgr2=0x7f86900cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8690150b80 0x7f86900cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 -- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c005280 msgr2=0x7f86901f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c005280 0x7f86901f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 -- 172.21.15.136:0/794882108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869014c370 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8695d10640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f868801cc00 tx=0x7f867c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688001a90 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86880a6e20 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688005070 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86900ceec0 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86901f3760 con 0x7f869014c570 2024-04-25T11:23:35.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.488+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f869006a350 con 0x7f869014c570 2024-04-25T11:23:35.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.492+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8688005210 con 0x7f869014c570 2024-04-25T11:23:35.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.492+0000 7f8685ffb640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 0x7f8674071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:35.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.492+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f8688128c30 con 0x7f869014c570 2024-04-25T11:23:35.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.492+0000 7f869550f640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 0x7f8674071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:35.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.492+0000 7f869550f640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 0x7f8674071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f868c009910 tx=0x7f868c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:35.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.496+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f869014c370 con 0x7f869014c570 2024-04-25T11:23:35.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.728+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f869014be70 con 0x7f869014c570 2024-04-25T11:23:35.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8685ffb640 1 -- 172.21.15.136:0/794882108 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6097 (secure 0 0 0) 0x7f8688016430 con 0x7f869014c570 2024-04-25T11:23:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:35.727 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:34.055100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:reconnect","state_seq":53,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:35.727 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 22 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 msgr2=0x7f8674071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 0x7f8674071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f868c009910 tx=0x7f868c008040 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 msgr2=0x7f869014b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f868801cc00 tx=0x7f867c005840 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 shutdown_connections 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f869014c570 0x7f869014b1f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f867406f430 0x7f8674071940 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8690150b80 0x7f86900cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 --2- 172.21.15.136:0/794882108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f867c005280 0x7f86901f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 >> 172.21.15.136:0/794882108 conn(0x7f8690070f70 msgr2=0x7f86900cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 shutdown_connections 2024-04-25T11:23:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:35.732+0000 7f8697773640 1 -- 172.21.15.136:0/794882108 wait complete. 2024-04-25T11:23:35.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:37.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:37.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 Processor -- start 2024-04-25T11:23:37.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 -- start start 2024-04-25T11:23:37.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9cdc0749a0 0x7f9cdc074db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9cdc0c4340 0x7f9cdc0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cdc0c3210 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9cdc15cae0 con 0x7f9cdc0749a0 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9ce28d8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9cdc0c9d20 con 0x7f9cdc0c4340 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.932+0000 7f9cd3fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9cdc0749a0 0x7f9cdc074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cdbfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9cdc0c2e00 0x7f9cdc15bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59110/0 (socket says 172.21.15.136:59110) 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9cdc0749a0 0x7f9cdc074db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46050/0 (socket says 172.21.15.136:46050) 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cdbfff640 1 -- 172.21.15.136:0/3147286060 learned_addr learned my addr 172.21.15.136:0/3147286060 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9ce0e75640 1 --2- 172.21.15.136:0/3147286060 >> v2:172.21.15.157:3300/0 conn(0x7f9cdc0c4340 0x7f9cdc0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 -- 172.21.15.136:0/3147286060 >> v2:172.21.15.157:3300/0 conn(0x7f9cdc0c4340 msgr2=0x7f9cdc0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:37.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 --2- 172.21.15.136:0/3147286060 >> v2:172.21.15.157:3300/0 conn(0x7f9cdc0c4340 0x7f9cdc0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:37.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 -- 172.21.15.136:0/3147286060 >> v1:172.21.15.136:6789/0 conn(0x7f9cdc0c2e00 legacy=0x7f9cdc15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:37.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 -- 172.21.15.136:0/3147286060 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc0c4c90 con 0x7f9cdc0749a0 2024-04-25T11:23:37.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cd3fff640 1 --2- 172.21.15.136:0/3147286060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc074db0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40097c0 tx=0x7f9cc401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b9ee6cccc1befb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:37.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cdb7fe640 1 -- 172.21.15.136:0/3147286060 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b0070 con 0x7f9cdc0749a0 2024-04-25T11:23:37.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cdb7fe640 1 -- 172.21.15.136:0/3147286060 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc40ab600 con 0x7f9cdc0749a0 2024-04-25T11:23:37.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.936+0000 7f9cdb7fe640 1 -- 172.21.15.136:0/3147286060 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40ab9d0 con 0x7f9cdc0749a0 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- 172.21.15.136:0/3147286060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0749a0 msgr2=0x7f9cdc074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/3147286060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc074db0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40097c0 tx=0x7f9cc401c340 comp rx=0 tx=0).stop 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- 172.21.15.136:0/3147286060 shutdown_connections 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/3147286060 >> v2:172.21.15.157:3300/0 conn(0x7f9cdc0c4340 0x7f9cdc0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/3147286060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc074db0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- 172.21.15.136:0/3147286060 >> 172.21.15.136:0/3147286060 conn(0x7f9cdc0ca5a0 msgr2=0x7f9cdc0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- 172.21.15.136:0/3147286060 shutdown_connections 2024-04-25T11:23:37.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- 172.21.15.136:0/3147286060 wait complete. 2024-04-25T11:23:37.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 Processor -- start 2024-04-25T11:23:37.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.940+0000 7f9ce28d8640 1 -- start start 2024-04-25T11:23:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 0x7f9cdc145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc15c4d0 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc0ca000 con 0x7f9cdc0749a0 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cdc073200 con 0x7f9cdc0c4340 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46058/0 (socket says 172.21.15.136:46058) 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 -- 172.21.15.136:0/2398738002 learned_addr learned my addr 172.21.15.136:0/2398738002 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cdbfff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce0e75640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 0x7f9cdc145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 -- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 msgr2=0x7f9cdc145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 0x7f9cdc145890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 -- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 msgr2=0x7f9cdc146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 -- 172.21.15.136:0/2398738002 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc147160 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cdbfff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce0e75640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 0x7f9cdc145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:37.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd3fff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40097c0 tx=0x7f9cc40190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b8030 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc40ab400 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b0040 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc1f2e20 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc146140 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc00052a0 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.944+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9cc40bb070 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.948+0000 7f9cd97fa640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 0x7f9cbc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:37.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.948+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f9cc4128dc0 con 0x7f9cdc0c2e00 2024-04-25T11:23:37.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.948+0000 7f9cdbfff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 0x7f9cbc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:37.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.948+0000 7f9cdbfff640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 0x7f9cbc071910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000f70 tx=0x7f9ccc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:37.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:37.952+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cc40c2070 con 0x7f9cdc0c2e00 2024-04-25T11:23:38.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.204+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc0005860 con 0x7f9cdc0c2e00 2024-04-25T11:23:38.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9cd97fa640 1 -- 172.21.15.136:0/2398738002 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6097 (secure 0 0 0) 0x7f9cc4016050 con 0x7f9cdc0c2e00 2024-04-25T11:23:38.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:38.203 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:34.055100+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:reconnect","state_seq":53,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:38.203 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 22 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 msgr2=0x7f9cbc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 0x7f9cbc071910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000f70 tx=0x7f9ccc009040 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 msgr2=0x7f9cdc145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40097c0 tx=0x7f9cc40190e0 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 shutdown_connections 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cbc06f400 0x7f9cbc071910 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cdc0c4340 0x7f9cdc145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cdc0c2e00 0x7f9cdc145350 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 --2- 172.21.15.136:0/2398738002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cdc0749a0 0x7f9cdc146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 >> 172.21.15.136:0/2398738002 conn(0x7f9cdc0ca5a0 msgr2=0x7f9cdc06d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 shutdown_connections 2024-04-25T11:23:38.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:38.208+0000 7f9ce28d8640 1 -- 172.21.15.136:0/2398738002 wait complete. 2024-04-25T11:23:38.228 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4254": { "addr": "172.21.15.136:6833/3353376405", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 3353376405, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 3353376405, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4254, "incarnation": 20, "join_fscid": -1, "name": "a", "rank": 4, "state": "up:reconnect", "state_seq": 53 }, "gid_4260": { "addr": "172.21.15.136:6837/1368493675", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1368493675, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1368493675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:23:34.055100+0000", "qdb_cluster": [ 4260, 4269, 4297, 4303 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 4260, "mds_4": 4254 } } } ], "standbys": [] } 2024-04-25T11:23:40.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 Processor -- start 2024-04-25T11:23:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 -- start start 2024-04-25T11:23:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4814074980 0x7f48140c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48140c3750 con 0x7f48140c4170 2024-04-25T11:23:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48140c3190 con 0x7f481414f840 2024-04-25T11:23:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481afc1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f481406dc00 con 0x7f4814074980 2024-04-25T11:23:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481955e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f48140c4170 0x7f48141636f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59126/0 (socket says 172.21.15.136:59126) 2024-04-25T11:23:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481955e640 1 -- 172.21.15.136:0/1288154365 learned_addr learned my addr 172.21.15.136:0/1288154365 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480bfff640 1 --2- 172.21.15.136:0/1288154365 >> v2:172.21.15.157:3300/0 conn(0x7f4814074980 0x7f48140c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190454769 0 0) 0x7f48140c3750 con 0x7f48140c4170 2024-04-25T11:23:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc003530 con 0x7f48140c4170 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1925660357 0 0) 0x7f47fc003530 con 0x7f48140c4170 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 >> v2:172.21.15.157:3300/0 conn(0x7f4814074980 msgr2=0x7f48140c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 --2- 172.21.15.136:0/1288154365 >> v2:172.21.15.157:3300/0 conn(0x7f4814074980 0x7f48140c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 >> v1:172.21.15.157:6789/0 conn(0x7f481414f840 legacy=0x7f4814073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48140c3470 con 0x7f48140c4170 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f480c001a00 con 0x7f48140c4170 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 >> v1:172.21.15.136:6789/0 conn(0x7f48140c4170 legacy=0x7f48141636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47fc0039c0 0x7f47fc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc004810 0x7f47fc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 0x7f47fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f481406dc00 con 0x7f47fc0051d0 2024-04-25T11:23:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00c170 con 0x7f47fc0039c0 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc00c370 con 0x7f47fc004810 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f481955e640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 0x7f47fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f4818d5d640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47fc0039c0 0x7f47fc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.424+0000 7f480bfff640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc004810 0x7f47fc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 -- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc004810 msgr2=0x7f47fc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc004810 0x7f47fc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 -- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47fc0039c0 msgr2=0x7f47fc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47fc0039c0 0x7f47fc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 -- 172.21.15.136:0/1288154365 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc003530 con 0x7f47fc0051d0 2024-04-25T11:23:40.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f4818d5d640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47fc0039c0 0x7f47fc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 -- 172.21.15.136:0/1288154365 reap_dead start 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481955e640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 0x7f47fc00b9d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f480c00a440 tx=0x7f480c01f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9ca321d6c23fea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c00a480 con 0x7f47fc0051d0 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c0af620 con 0x7f47fc0051d0 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f480b7fe640 1 -- 172.21.15.136:0/1288154365 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c0afa10 con 0x7f47fc0051d0 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 -- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 msgr2=0x7f47fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 0x7f47fc00b9d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f480c00a440 tx=0x7f480c01f4b0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 -- 172.21.15.136:0/1288154365 shutdown_connections 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 --2- 172.21.15.136:0/1288154365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47fc0051d0 0x7f47fc00b9d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 -- 172.21.15.136:0/1288154365 >> 172.21.15.136:0/1288154365 conn(0x7f48140ca5a0 msgr2=0x7f48140ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:40.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 -- 172.21.15.136:0/1288154365 shutdown_connections 2024-04-25T11:23:40.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.428+0000 7f481afc1640 1 -- 172.21.15.136:0/1288154365 wait complete. 2024-04-25T11:23:40.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.432+0000 7f481afc1640 1 Processor -- start 2024-04-25T11:23:40.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.432+0000 7f481afc1640 1 -- start start 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc0051d0 0x7f48140b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48140c8e00 0x7f48140cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f48140c4580 con 0x7f4814144fe0 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f481414fc50 con 0x7f48140c8e00 2024-04-25T11:23:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4814074d90 con 0x7f47fc0051d0 2024-04-25T11:23:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46066/0 (socket says 172.21.15.136:46066) 2024-04-25T11:23:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 -- 172.21.15.136:0/3154935873 learned_addr learned my addr 172.21.15.136:0/3154935873 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f4818d5d640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc0051d0 0x7f48140b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481955e640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48140c8e00 0x7f48140cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 -- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc0051d0 msgr2=0x7f48140b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc0051d0 0x7f48140b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 -- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48140c8e00 msgr2=0x7f48140cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48140c8e00 0x7f48140cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 -- 172.21.15.136:0/3154935873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48140706b0 con 0x7f4814144fe0 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f480bfff640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4804011590 tx=0x7f4804011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804015870 con 0x7f4814144fe0 2024-04-25T11:23:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4804015a10 con 0x7f4814144fe0 2024-04-25T11:23:40.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4814157690 con 0x7f4814144fe0 2024-04-25T11:23:40.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804015dc0 con 0x7f4814144fe0 2024-04-25T11:23:40.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.436+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4814157c50 con 0x7f4814144fe0 2024-04-25T11:23:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4814157e90 con 0x7f4814144fe0 2024-04-25T11:23:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4804013400 con 0x7f4814144fe0 2024-04-25T11:23:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f48097fa640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 0x7f4800071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f480408fd50 con 0x7f4814144fe0 2024-04-25T11:23:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f4818d5d640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 0x7f4800071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:40.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.440+0000 7f4818d5d640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 0x7f4800071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4810000c10 tx=0x7f4810009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:40.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.448+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4814157e90 con 0x7f4814144fe0 2024-04-25T11:23:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.668+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f481406a350 con 0x7f4814144fe0 2024-04-25T11:23:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.668+0000 7f48097fa640 1 -- 172.21.15.136:0/3154935873 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6094 (secure 0 0 0) 0x7f4804093030 con 0x7f4814144fe0 2024-04-25T11:23:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:40.666 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:39.070211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:rejoin","state_seq":55,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:23:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 23 2024-04-25T11:23:40.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 msgr2=0x7f4800071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 0x7f4800071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4810000c10 tx=0x7f4810009040 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 msgr2=0x7f48140c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4804011590 tx=0x7f4804011a50 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 shutdown_connections 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48140c8e00 0x7f48140cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4814144fe0 0x7f48140c88c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f480006f430 0x7f4800071940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 --2- 172.21.15.136:0/3154935873 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47fc0051d0 0x7f48140b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 >> 172.21.15.136:0/3154935873 conn(0x7f48140ca5a0 msgr2=0x7f481414e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 shutdown_connections 2024-04-25T11:23:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:40.672+0000 7f481afc1640 1 -- 172.21.15.136:0/3154935873 wait complete. 2024-04-25T11:23:40.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:42.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:42.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 Processor -- start 2024-04-25T11:23:42.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 -- start start 2024-04-25T11:23:42.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14800c96e0 0x7f14801540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14800c79e0 0x7f14800c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14800c9af0 con 0x7f148015a3d0 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f148006cc10 con 0x7f14800c96e0 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148856f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14800cb220 con 0x7f14800c79e0 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.848+0000 7f148630b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14800c96e0 0x7f14801540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14800c96e0 0x7f14801540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:34030/0 (socket says 172.21.15.136:34030) 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 -- 172.21.15.136:0/772699145 learned_addr learned my addr 172.21.15.136:0/772699145 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:42.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f1485b0a640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14800c79e0 0x7f14800c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 -- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14800c79e0 msgr2=0x7f14800c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:42.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14800c79e0 0x7f14800c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:42.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 -- 172.21.15.136:0/772699145 >> v1:172.21.15.157:6789/0 conn(0x7f148015a3d0 legacy=0x7f148015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:42.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 -- 172.21.15.136:0/772699145 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14800cb500 con 0x7f14800c96e0 2024-04-25T11:23:42.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f148630b640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c96e0 0x7f14801540d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f147c009870 tx=0x7f147c01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35ce36551369ee0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:42.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f1485309640 1 -- 172.21.15.136:0/772699145 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c0b5070 con 0x7f14800c96e0 2024-04-25T11:23:42.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f1485309640 1 -- 172.21.15.136:0/772699145 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147c0a8a50 con 0x7f14800c96e0 2024-04-25T11:23:42.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.852+0000 7f1485309640 1 -- 172.21.15.136:0/772699145 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c0a8e20 con 0x7f14800c96e0 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c96e0 msgr2=0x7f14801540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c96e0 0x7f14801540d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f147c009870 tx=0x7f147c01ed50 comp rx=0 tx=0).stop 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- 172.21.15.136:0/772699145 shutdown_connections 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14800c79e0 0x7f14800c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 --2- 172.21.15.136:0/772699145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c96e0 0x7f14801540d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- 172.21.15.136:0/772699145 >> 172.21.15.136:0/772699145 conn(0x7f148006f9d0 msgr2=0x7f148006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- 172.21.15.136:0/772699145 shutdown_connections 2024-04-25T11:23:42.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- 172.21.15.136:0/772699145 wait complete. 2024-04-25T11:23:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 Processor -- start 2024-04-25T11:23:42.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.856+0000 7f148856f640 1 -- start start 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c79e0 0x7f14801f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f148015a3d0 0x7f1480150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f14800cf0f0 con 0x7f14800c96e0 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1480154610 con 0x7f14800c79e0 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14800ced40 con 0x7f148015a3d0 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46078/0 (socket says 172.21.15.136:46078) 2024-04-25T11:23:42.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 -- 172.21.15.136:0/2326738123 learned_addr learned my addr 172.21.15.136:0/2326738123 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1486b0c640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f148015a3d0 0x7f1480150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148630b640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c79e0 0x7f14801f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 -- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f148015a3d0 msgr2=0x7f1480150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f148015a3d0 0x7f1480150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 -- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c79e0 msgr2=0x7f14801f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c79e0 0x7f14801f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 -- 172.21.15.136:0/2326738123 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14801f00b0 con 0x7f14800c96e0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1485b0a640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f147000e8a0 tx=0x7f147000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470017070 con 0x7f14800c96e0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147000c940 con 0x7f14800c96e0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148015cdd0 con 0x7f14800c96e0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147000ccf0 con 0x7f14800c96e0 2024-04-25T11:23:42.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14801f0610 con 0x7f14800c96e0 2024-04-25T11:23:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f148006a350 con 0x7f14800c96e0 2024-04-25T11:23:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f147001a070 con 0x7f14800c96e0 2024-04-25T11:23:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 0x7f1464071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.860+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f147008f610 con 0x7f14800c96e0 2024-04-25T11:23:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.864+0000 7f148630b640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 0x7f1464071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:42.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.864+0000 7f148630b640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 0x7f1464071910 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f147c0b2000 tx=0x7f147c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:42.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:42.868+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f14801f00b0 con 0x7f14800c96e0 2024-04-25T11:23:43.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.112+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f148015b5f0 con 0x7f14800c96e0 2024-04-25T11:23:43.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.112+0000 7f1476ffd640 1 -- 172.21.15.136:0/2326738123 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6094 (secure 0 0 0) 0x7f147005c0a0 con 0x7f14800c96e0 2024-04-25T11:23:43.108 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:43.109 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:39.070211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:rejoin","state_seq":55,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:23:43.109 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 23 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 msgr2=0x7f1464071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 0x7f1464071910 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f147c0b2000 tx=0x7f147c008040 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 msgr2=0x7f148015c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f147000e8a0 tx=0x7f147000eca0 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 shutdown_connections 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146406f400 0x7f1464071910 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f148015a3d0 0x7f1480150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14800c96e0 0x7f148015c890 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 --2- 172.21.15.136:0/2326738123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14800c79e0 0x7f14801f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 >> 172.21.15.136:0/2326738123 conn(0x7f148006f9d0 msgr2=0x7f14800c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 shutdown_connections 2024-04-25T11:23:43.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:43.116+0000 7f148856f640 1 -- 172.21.15.136:0/2326738123 wait complete. 2024-04-25T11:23:43.133 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:23:45.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:45.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 Processor -- start 2024-04-25T11:23:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 -- start start 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f42d40cdee0 0x7f42d40ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f42d40b5490 0x7f42d40b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f42d41602f0 con 0x7f42d40cdee0 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f42d415d010 con 0x7f42d40b5490 2024-04-25T11:23:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d988d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42d40ccb80 con 0x7f42d4156330 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f42d40cdee0 0x7f42d40ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f42d40cdee0 0x7f42d40ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55692/0 (socket says 172.21.15.136:55692) 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 -- 172.21.15.136:0/236908811 learned_addr learned my addr 172.21.15.136:0/236908811 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d2ffd640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42d37fe640 1 --2- 172.21.15.136:0/236908811 >> v2:172.21.15.157:3300/0 conn(0x7f42d40b5490 0x7f42d40b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 -- 172.21.15.136:0/236908811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 msgr2=0x7f42d415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 -- 172.21.15.136:0/236908811 >> v2:172.21.15.157:3300/0 conn(0x7f42d40b5490 msgr2=0x7f42d40b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 --2- 172.21.15.136:0/236908811 >> v2:172.21.15.157:3300/0 conn(0x7f42d40b5490 0x7f42d40b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.288+0000 7f42cbfff640 1 -- 172.21.15.136:0/236908811 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d4156740 con 0x7f42d40cdee0 2024-04-25T11:23:45.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42cbfff640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40cdee0 0x7f42d40ce2f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f42d40b70f0 tx=0x7f42bc027520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ff2b9ebb1603a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:45.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d27fc640 1 -- 172.21.15.136:0/236908811 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0bc070 con 0x7f42d40cdee0 2024-04-25T11:23:45.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d27fc640 1 -- 172.21.15.136:0/236908811 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc0b7600 con 0x7f42d40cdee0 2024-04-25T11:23:45.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d27fc640 1 -- 172.21.15.136:0/236908811 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0b79d0 con 0x7f42d40cdee0 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 -- 172.21.15.136:0/236908811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40cdee0 msgr2=0x7f42d40ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40cdee0 0x7f42d40ce2f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f42d40b70f0 tx=0x7f42bc027520 comp rx=0 tx=0).stop 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 -- 172.21.15.136:0/236908811 shutdown_connections 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40cdee0 0x7f42d40ce2f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 --2- 172.21.15.136:0/236908811 >> v2:172.21.15.157:3300/0 conn(0x7f42d40b5490 0x7f42d40b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 --2- 172.21.15.136:0/236908811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d415cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 -- 172.21.15.136:0/236908811 >> 172.21.15.136:0/236908811 conn(0x7f42d406d6c0 msgr2=0x7f42d406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 -- 172.21.15.136:0/236908811 shutdown_connections 2024-04-25T11:23:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.292+0000 7f42d988d640 1 -- 172.21.15.136:0/236908811 wait complete. 2024-04-25T11:23:45.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 Processor -- start 2024-04-25T11:23:45.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 -- start start 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 0x7f42d40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d41549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f42d415fca0 con 0x7f42d40b5490 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f42d415f9c0 con 0x7f42d40cdee0 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42d415ff80 con 0x7f42d4156330 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55708/0 (socket says 172.21.15.136:55708) 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 -- 172.21.15.136:0/1478479402 learned_addr learned my addr 172.21.15.136:0/1478479402 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42cbfff640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 0x7f42d40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d37fe640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d41549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 -- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 msgr2=0x7f42d41549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d41549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 -- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 msgr2=0x7f42d40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 0x7f42d40d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 -- 172.21.15.136:0/1478479402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d4151610 con 0x7f42d40b5490 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42cbfff640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 0x7f42d40d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d37fe640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d41549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d2ffd640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f42c400e870 tx=0x7f42c400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4017070 con 0x7f42d40b5490 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c400c8e0 con 0x7f42d40b5490 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d40d08b0 con 0x7f42d40b5490 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.296+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c400cc90 con 0x7f42d40b5490 2024-04-25T11:23:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d40cb410 con 0x7f42d40b5490 2024-04-25T11:23:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d406a350 con 0x7f42d40b5490 2024-04-25T11:23:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f42c401a070 con 0x7f42d40b5490 2024-04-25T11:23:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42d0ff9640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 0x7f42b4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:45.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f42c408f530 con 0x7f42d40b5490 2024-04-25T11:23:45.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42cbfff640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 0x7f42b4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:45.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.300+0000 7f42cbfff640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 0x7f42b4071940 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0b9000 tx=0x7f42bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:45.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.304+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f42c405c050 con 0x7f42d40b5490 2024-04-25T11:23:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42d40c7300 con 0x7f42d40b5490 2024-04-25T11:23:45.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d0ff9640 1 -- 172.21.15.136:0/1478479402 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6099 (secure 0 0 0) 0x7f42c405b950 con 0x7f42d40b5490 2024-04-25T11:23:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:45.535 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4254,4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:23:45.535 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 24 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 msgr2=0x7f42b4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 0x7f42b4071940 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0b9000 tx=0x7f42bc008040 comp rx=0 tx=0).stop 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 msgr2=0x7f42d40ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f42c400e870 tx=0x7f42c400ec70 comp rx=0 tx=0).stop 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 shutdown_connections 2024-04-25T11:23:45.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f42b406f430 0x7f42b4071940 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f42d4156330 0x7f42d41549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42d40cdee0 0x7f42d40d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 --2- 172.21.15.136:0/1478479402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d40b5490 0x7f42d40ca830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 >> 172.21.15.136:0/1478479402 conn(0x7f42d406d6c0 msgr2=0x7f42d40b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 shutdown_connections 2024-04-25T11:23:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:45.540+0000 7f42d988d640 1 -- 172.21.15.136:0/1478479402 wait complete. 2024-04-25T11:23:45.558 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=4, replacing gid=4257 2024-04-25T11:23:45.558 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2024-04-25T11:23:48.474 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:23:48.475+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:23:49.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:23:49.987 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:23:54.559 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-25T11:23:54.560 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T11:23:54.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T11:23:54.562 INFO:tasks.ceph.mds.c:Started 2024-04-25T11:23:54.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:23:54.673 INFO:tasks.ceph.mds.c.smithi136.stdout:starting mds.c at 2024-04-25T11:23:54.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 Processor -- start 2024-04-25T11:23:54.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 -- start start 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5ee80c96e0 0x7f5ee81540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ee80c9af0 con 0x7f5ee80c79e0 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee806cc10 con 0x7f5ee80c96e0 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5ef05ce640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee80cb220 con 0x7f5ee815aae0 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eee36a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5ee80c96e0 0x7f5ee81540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eedb69640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5ee80c79e0 0x7f5ee815a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48014/0 (socket says 172.21.15.136:48014) 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eedb69640 1 -- 172.21.15.136:0/3837363435 learned_addr learned my addr 172.21.15.136:0/3837363435 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eeeb6b640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eee36a640 1 -- 172.21.15.136:0/3837363435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 msgr2=0x7f5ee80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eee36a640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eee36a640 1 -- 172.21.15.136:0/3837363435 >> v1:172.21.15.136:6789/0 conn(0x7f5ee80c79e0 legacy=0x7f5ee815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.744+0000 7f5eee36a640 1 -- 172.21.15.136:0/3837363435 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee80cb500 con 0x7f5ee80c96e0 2024-04-25T11:23:54.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5eee36a640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c96e0 0x7f5ee81540d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097c0 tx=0x7f5ee401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710ab082256b47c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:54.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5eed368640 1 -- 172.21.15.136:0/3837363435 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40b5070 con 0x7f5ee80c96e0 2024-04-25T11:23:54.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5eed368640 1 -- 172.21.15.136:0/3837363435 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee40a8990 con 0x7f5ee80c96e0 2024-04-25T11:23:54.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5eed368640 1 -- 172.21.15.136:0/3837363435 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40a8d60 con 0x7f5ee80c96e0 2024-04-25T11:23:54.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 -- 172.21.15.136:0/3837363435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c96e0 msgr2=0x7f5ee81540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c96e0 0x7f5ee81540d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40097c0 tx=0x7f5ee401ec90 comp rx=0 tx=0).stop 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 -- 172.21.15.136:0/3837363435 shutdown_connections 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/3837363435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c96e0 0x7f5ee81540d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 -- 172.21.15.136:0/3837363435 >> 172.21.15.136:0/3837363435 conn(0x7f5ee806f9d0 msgr2=0x7f5ee806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 -- 172.21.15.136:0/3837363435 shutdown_connections 2024-04-25T11:23:54.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.748+0000 7f5ef05ce640 1 -- 172.21.15.136:0/3837363435 wait complete. 2024-04-25T11:23:54.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 Processor -- start 2024-04-25T11:23:54.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 -- start start 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 0x7f5ee8154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee8156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee80cf0f0 con 0x7f5ee80c79e0 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee8154610 con 0x7f5ee80c96e0 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5ef05ce640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee80ced40 con 0x7f5ee815aae0 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48938/0 (socket says 172.21.15.136:48938) 2024-04-25T11:23:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 -- 172.21.15.136:0/2992254731 learned_addr learned my addr 172.21.15.136:0/2992254731 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eeeb6b640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee8156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eedb69640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 0x7f5ee8154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 -- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 msgr2=0x7f5ee8156a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee8156a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 -- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 msgr2=0x7f5ee8154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 0x7f5ee8154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 -- 172.21.15.136:0/2992254731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee8151470 con 0x7f5ee80c79e0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eedb69640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 0x7f5ee8154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5eee36a640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40b3040 tx=0x7f5ee401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40bf070 con 0x7f5ee80c79e0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee40b07a0 con 0x7f5ee80c79e0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.752+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40b5040 con 0x7f5ee80c79e0 2024-04-25T11:23:54.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee8151a10 con 0x7f5ee80c79e0 2024-04-25T11:23:54.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee8156240 con 0x7f5ee80c79e0 2024-04-25T11:23:54.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee806a350 con 0x7f5ee80c79e0 2024-04-25T11:23:54.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ee40b0b90 con 0x7f5ee80c79e0 2024-04-25T11:23:54.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5edeffd640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 0x7f5ecc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:23:54.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f5ee412bf70 con 0x7f5ee80c79e0 2024-04-25T11:23:54.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5eedb69640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 0x7f5ecc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:23:54.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.756+0000 7f5eedb69640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 0x7f5ecc071940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5ee806e630 tx=0x7f5ed8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:23:54.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.764+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ee40187f0 con 0x7f5ee80c79e0 2024-04-25T11:23:54.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.980+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ee814f860 con 0x7f5ee80c79e0 2024-04-25T11:23:54.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.980+0000 7f5edeffd640 1 -- 172.21.15.136:0/2992254731 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f5ee40f8a90 con 0x7f5ee80c79e0 2024-04-25T11:23:54.976 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:23:54.977 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:23:54.977 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 msgr2=0x7f5ecc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 0x7f5ecc071940 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5ee806e630 tx=0x7f5ed8009040 comp rx=0 tx=0).stop 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 msgr2=0x7f5ee81f4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40b3040 tx=0x7f5ee401ba30 comp rx=0 tx=0).stop 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 shutdown_connections 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ecc06f430 0x7f5ecc071940 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ee815aae0 0x7f5ee8156a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ee80c96e0 0x7f5ee8154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 --2- 172.21.15.136:0/2992254731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ee80c79e0 0x7f5ee81f4bf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 >> 172.21.15.136:0/2992254731 conn(0x7f5ee806f9d0 msgr2=0x7f5ee80c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 shutdown_connections 2024-04-25T11:23:54.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:23:54.984+0000 7f5ef05ce640 1 -- 172.21.15.136:0/2992254731 wait complete. 2024-04-25T11:23:54.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-25T11:23:54.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2024-04-25T11:24:20.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:24:20.227 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:24:50.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:24:50.342 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:25:20.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:25:20.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:25:40.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:40.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.178+0000 7fce3925f640 1 Processor -- start 2024-04-25T11:25:40.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.178+0000 7fce3925f640 1 -- start start 2024-04-25T11:25:40.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3925f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce340c79e0 0x7fce340c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3925f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fce3415a3d0 0x7fce3415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3925f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce340c9af0 con 0x7fce340c96e0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3925f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fce3406cc10 con 0x7fce340c79e0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3925f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fce340cb220 con 0x7fce3415a3d0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce2bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce340c79e0 0x7fce340c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce32d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fce340c96e0 0x7fce341540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:35340/0 (socket says 172.21.15.136:35340) 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce32d9d640 1 -- 172.21.15.136:0/3291647456 learned_addr learned my addr 172.21.15.136:0/3291647456 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce2bfff640 1 -- 172.21.15.136:0/3291647456 >> v2:172.21.15.157:3300/0 conn(0x7fce3415a3d0 msgr2=0x7fce3415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3359e640 1 --2- 172.21.15.136:0/3291647456 >> v2:172.21.15.157:3300/0 conn(0x7fce3415a3d0 0x7fce3415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce2bfff640 1 --2- 172.21.15.136:0/3291647456 >> v2:172.21.15.157:3300/0 conn(0x7fce3415a3d0 0x7fce3415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce2bfff640 1 -- 172.21.15.136:0/3291647456 >> v1:172.21.15.136:6789/0 conn(0x7fce340c96e0 legacy=0x7fce341540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce3259c640 1 -- 172.21.15.136:0/3291647456 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428488219 0 0) 0x7fce340c9af0 con 0x7fce340c96e0 2024-04-25T11:25:40.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.182+0000 7fce2bfff640 1 -- 172.21.15.136:0/3291647456 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce340cb500 con 0x7fce340c79e0 2024-04-25T11:25:40.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce2bfff640 1 --2- 172.21.15.136:0/3291647456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c79e0 0x7fce340c7df0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fce34075140 tx=0x7fce1c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9034dde93cd8bfd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3259c640 1 -- 172.21.15.136:0/3291647456 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0b0070 con 0x7fce340c79e0 2024-04-25T11:25:40.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3259c640 1 -- 172.21.15.136:0/3291647456 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1c0ab600 con 0x7fce340c79e0 2024-04-25T11:25:40.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3259c640 1 -- 172.21.15.136:0/3291647456 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0ab9d0 con 0x7fce340c79e0 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- 172.21.15.136:0/3291647456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c79e0 msgr2=0x7fce340c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 --2- 172.21.15.136:0/3291647456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c79e0 0x7fce340c7df0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fce34075140 tx=0x7fce1c01c480 comp rx=0 tx=0).stop 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- 172.21.15.136:0/3291647456 shutdown_connections 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 --2- 172.21.15.136:0/3291647456 >> v2:172.21.15.157:3300/0 conn(0x7fce3415a3d0 0x7fce3415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 --2- 172.21.15.136:0/3291647456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c79e0 0x7fce340c7df0 secure :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fce34075140 tx=0x7fce1c01c480 comp rx=0 tx=0).stop 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- 172.21.15.136:0/3291647456 >> 172.21.15.136:0/3291647456 conn(0x7fce3406f9d0 msgr2=0x7fce3406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- 172.21.15.136:0/3291647456 shutdown_connections 2024-04-25T11:25:40.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- 172.21.15.136:0/3291647456 wait complete. 2024-04-25T11:25:40.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 Processor -- start 2024-04-25T11:25:40.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.186+0000 7fce3925f640 1 -- start start 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce3415a3d0 0x7fce34143950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce34140c60 0x7fce34141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce340ced40 con 0x7fce340c96e0 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce3406d440 con 0x7fce3415a3d0 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce341f86b0 con 0x7fce34140c60 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56846/0 (socket says 172.21.15.136:56846) 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 -- 172.21.15.136:0/2168334707 learned_addr learned my addr 172.21.15.136:0/2168334707 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce2bfff640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce3415a3d0 0x7fce34143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3359e640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce34140c60 0x7fce34141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 -- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce34140c60 msgr2=0x7fce34141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce34140c60 0x7fce34141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 -- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce3415a3d0 msgr2=0x7fce34143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce3415a3d0 0x7fce34143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 -- 172.21.15.136:0/2168334707 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce341632c0 con 0x7fce340c96e0 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce32d9d640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fce2400c4b0 tx=0x7fce2400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24018070 con 0x7fce340c96e0 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce24007500 con 0x7fce340c96e0 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce341635a0 con 0x7fce340c96e0 2024-04-25T11:25:40.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24004a30 con 0x7fce340c96e0 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce34145a40 con 0x7fce340c96e0 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce34145580 con 0x7fce340c96e0 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.190+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fce2401b070 con 0x7fce340c96e0 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.194+0000 7fce2b7fe640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 0x7fce14071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.194+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fce2408e950 con 0x7fce340c96e0 2024-04-25T11:25:40.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.194+0000 7fce2bfff640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 0x7fce14071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.194+0000 7fce2bfff640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 0x7fce14071940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0ad000 tx=0x7fce1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.198+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce34145580 con 0x7fce340c96e0 2024-04-25T11:25:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.434+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce3406a350 con 0x7fce340c96e0 2024-04-25T11:25:40.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce2b7fe640 1 -- 172.21.15.136:0/2168334707 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7fce2405b3e0 con 0x7fce340c96e0 2024-04-25T11:25:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:40.432 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:40.432 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 msgr2=0x7fce14071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 0x7fce14071940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0ad000 tx=0x7fce1c008040 comp rx=0 tx=0).stop 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 msgr2=0x7fce34145f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fce2400c4b0 tx=0x7fce2400c970 comp rx=0 tx=0).stop 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 shutdown_connections 2024-04-25T11:25:40.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce1406f430 0x7fce14071940 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce34140c60 0x7fce34141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce3415a3d0 0x7fce34143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 --2- 172.21.15.136:0/2168334707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340c96e0 0x7fce34145f60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 >> 172.21.15.136:0/2168334707 conn(0x7fce3406f9d0 msgr2=0x7fce341473a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 shutdown_connections 2024-04-25T11:25:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.438+0000 7fce3925f640 1 -- 172.21.15.136:0/2168334707 wait complete. 2024-04-25T11:25:40.455 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=4) 2024-04-25T11:25:40.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T11:25:40.455 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:25:40.470 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T11:25:40.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 Processor -- start 2024-04-25T11:25:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 -- start start 2024-04-25T11:25:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9980154060 0x7f99800c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99800c8d40 con 0x7f998015aa00 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99800c3880 con 0x7f998015a5f0 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f9984b58640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f99800c8fd0 con 0x7f9980154060 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9980154060 0x7f99800c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997ed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f998015aa00 0x7f9980164700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:35352/0 (socket says 172.21.15.136:35352) 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997ed9d640 1 -- 172.21.15.136:0/2639034054 learned_addr learned my addr 172.21.15.136:0/2639034054 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997e59c640 1 -- 172.21.15.136:0/2639034054 >> v1:172.21.15.157:6789/0 conn(0x7f998015a5f0 legacy=0x7f99800c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997e59c640 1 -- 172.21.15.136:0/2639034054 >> v1:172.21.15.136:6789/0 conn(0x7f998015aa00 legacy=0x7f9980164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997d59a640 1 -- 172.21.15.136:0/2639034054 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628688729 0 0) 0x7f99800c8d40 con 0x7f998015aa00 2024-04-25T11:25:40.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.670+0000 7f997e59c640 1 -- 172.21.15.136:0/2639034054 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f998006d9b0 con 0x7f9980154060 2024-04-25T11:25:40.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f997e59c640 1 --2- 172.21.15.136:0/2639034054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99800c8800 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f99700097c0 tx=0x7f997001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c21743da05a26d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f997d59a640 1 -- 172.21.15.136:0/2639034054 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99700b5070 con 0x7f9980154060 2024-04-25T11:25:40.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f997d59a640 1 -- 172.21.15.136:0/2639034054 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99700a8990 con 0x7f9980154060 2024-04-25T11:25:40.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f997d59a640 1 -- 172.21.15.136:0/2639034054 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99700a8d60 con 0x7f9980154060 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 -- 172.21.15.136:0/2639034054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 msgr2=0x7f99800c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 --2- 172.21.15.136:0/2639034054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99800c8800 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f99700097c0 tx=0x7f997001ec90 comp rx=0 tx=0).stop 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 -- 172.21.15.136:0/2639034054 shutdown_connections 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 --2- 172.21.15.136:0/2639034054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99800c8800 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 -- 172.21.15.136:0/2639034054 >> 172.21.15.136:0/2639034054 conn(0x7f99800ca5a0 msgr2=0x7f99800ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 -- 172.21.15.136:0/2639034054 shutdown_connections 2024-04-25T11:25:40.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.674+0000 7f9984b58640 1 -- 172.21.15.136:0/2639034054 wait complete. 2024-04-25T11:25:40.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 Processor -- start 2024-04-25T11:25:40.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- start start 2024-04-25T11:25:40.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f998015a5f0 0x7f9980146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f998015aa00 0x7f99801592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f998006df30 con 0x7f9980154060 2024-04-25T11:25:40.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9980154470 con 0x7f998015aa00 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f998015a0d0 con 0x7f998015a5f0 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56864/0 (socket says 172.21.15.136:56864) 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 -- 172.21.15.136:0/2887670102 learned_addr learned my addr 172.21.15.136:0/2887670102 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997dd9b640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f998015a5f0 0x7f9980146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997ed9d640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f998015aa00 0x7f99801592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 -- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f998015a5f0 msgr2=0x7f9980146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f998015a5f0 0x7f9980146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 -- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f998015aa00 msgr2=0x7f99801592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f998015aa00 0x7f99801592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 -- 172.21.15.136:0/2887670102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99801588b0 con 0x7f9980154060 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f997e59c640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f99700097c0 tx=0x7f997001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99700b5040 con 0x7f9980154060 2024-04-25T11:25:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99700bf070 con 0x7f9980154060 2024-04-25T11:25:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99700b0aa0 con 0x7f9980154060 2024-04-25T11:25:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f998014ae40 con 0x7f9980154060 2024-04-25T11:25:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.678+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99801f0420 con 0x7f9980154060 2024-04-25T11:25:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f998006a350 con 0x7f9980154060 2024-04-25T11:25:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f99700b0c40 con 0x7f9980154060 2024-04-25T11:25:40.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f996effd640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 0x7f9960071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:40.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f997012d1b0 con 0x7f9980154060 2024-04-25T11:25:40.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f997dd9b640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 0x7f9960071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:40.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.682+0000 7f997dd9b640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 0x7f9960071910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f998015e1e0 tx=0x7f9974009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:40.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.686+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f99700ca060 con 0x7f9980154060 2024-04-25T11:25:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.926+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99801576a0 con 0x7f9980154060 2024-04-25T11:25:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.926+0000 7f996effd640 1 -- 172.21.15.136:0/2887670102 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f99700187f0 con 0x7f9980154060 2024-04-25T11:25:40.922 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:40.922 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 msgr2=0x7f9960071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 0x7f9960071910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f998015e1e0 tx=0x7f9974009040 comp rx=0 tx=0).stop 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 msgr2=0x7f99801f0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f99700097c0 tx=0x7f997001ba30 comp rx=0 tx=0).stop 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 shutdown_connections 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f998015aa00 0x7f99801592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f996006f400 0x7f9960071910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f998015a5f0 0x7f9980146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 --2- 172.21.15.136:0/2887670102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9980154060 0x7f99801f0f00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 >> 172.21.15.136:0/2887670102 conn(0x7f99800ca5a0 msgr2=0x7f9980070e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 shutdown_connections 2024-04-25T11:25:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:40.930+0000 7f9984b58640 1 -- 172.21.15.136:0/2887670102 wait complete. 2024-04-25T11:25:40.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:42.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.114+0000 7f4127b0c640 1 Processor -- start 2024-04-25T11:25:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.114+0000 7f4127b0c640 1 -- start start 2024-04-25T11:25:43.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4120156750 0x7f41200ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 0x7f41200cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f412006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f412015c410 con 0x7f4120156750 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f41200cfde0 con 0x7f41200ca7d0 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f4127b0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f412015c6b0 con 0x7f41200cf9d0 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4120156750 0x7f41200ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4120156750 0x7f41200ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56872/0 (socket says 172.21.15.136:56872) 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 -- 172.21.15.136:0/2562739351 learned_addr learned my addr 172.21.15.136:0/2562739351 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41260a9640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f412006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41250a7640 1 --2- 172.21.15.136:0/2562739351 >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 0x7f41200cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 -- 172.21.15.136:0/2562739351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 msgr2=0x7f412006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f412006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 -- 172.21.15.136:0/2562739351 >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 msgr2=0x7f41200cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 --2- 172.21.15.136:0/2562739351 >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 0x7f41200cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 -- 172.21.15.136:0/2562739351 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412015c990 con 0x7f4120156750 2024-04-25T11:25:43.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41250a7640 1 --2- 172.21.15.136:0/2562739351 >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 0x7f41200cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:25:43.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41260a9640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f412006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:25:43.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.118+0000 7f41258a8640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4120156750 0x7f41200ca290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f412006f960 tx=0x7f411c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0e61eecc4972d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f41248a6640 1 -- 172.21.15.136:0/2562739351 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0b0070 con 0x7f4120156750 2024-04-25T11:25:43.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f41248a6640 1 -- 172.21.15.136:0/2562739351 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c0ab600 con 0x7f4120156750 2024-04-25T11:25:43.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f41248a6640 1 -- 172.21.15.136:0/2562739351 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0ab9d0 con 0x7f4120156750 2024-04-25T11:25:43.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- 172.21.15.136:0/2562739351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4120156750 msgr2=0x7f41200ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4120156750 0x7f41200ca290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f412006f960 tx=0x7f411c01c520 comp rx=0 tx=0).stop 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- 172.21.15.136:0/2562739351 shutdown_connections 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f412006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 --2- 172.21.15.136:0/2562739351 >> v2:172.21.15.157:3300/0 conn(0x7f41200ca7d0 0x7f41200cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 --2- 172.21.15.136:0/2562739351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4120156750 0x7f41200ca290 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- 172.21.15.136:0/2562739351 >> 172.21.15.136:0/2562739351 conn(0x7f4120070f70 msgr2=0x7f4120068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- 172.21.15.136:0/2562739351 shutdown_connections 2024-04-25T11:25:43.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- 172.21.15.136:0/2562739351 wait complete. 2024-04-25T11:25:43.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 Processor -- start 2024-04-25T11:25:43.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.122+0000 7f4127b0c640 1 -- start start 2024-04-25T11:25:43.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f4120150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41200b0530 0x7f4120148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f412006d9e0 con 0x7f41200ca7d0 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f412006d860 con 0x7f41200b0530 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f412015ff80 con 0x7f41200cf9d0 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56874/0 (socket says 172.21.15.136:56874) 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 -- 172.21.15.136:0/1460644334 learned_addr learned my addr 172.21.15.136:0/1460644334 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41250a7640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f4120150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 -- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 msgr2=0x7f4120150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f4120150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 -- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41200b0530 msgr2=0x7f4120148e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:25:43.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41200b0530 0x7f4120148e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 -- 172.21.15.136:0/1460644334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41201f9590 con 0x7f41200ca7d0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41258a8640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f411c0ae040 tx=0x7f411c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0ba070 con 0x7f41200ca7d0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411c0b4620 con 0x7f41200ca7d0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c0b0040 con 0x7f41200ca7d0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41200ceaf0 con 0x7f41200ca7d0 2024-04-25T11:25:43.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41200cf070 con 0x7f41200ca7d0 2024-04-25T11:25:43.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412006a350 con 0x7f41200ca7d0 2024-04-25T11:25:43.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f411c0bd070 con 0x7f41200ca7d0 2024-04-25T11:25:43.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 0x7f4104071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:43.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f411c128920 con 0x7f41200ca7d0 2024-04-25T11:25:43.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.126+0000 7f41250a7640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 0x7f4104071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:43.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.130+0000 7f41250a7640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 0x7f4104071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4120151160 tx=0x7f4110008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:43.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.134+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f411c016110 con 0x7f41200ca7d0 2024-04-25T11:25:43.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.358+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41200cbf50 con 0x7f41200ca7d0 2024-04-25T11:25:43.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.358+0000 7f41167fc640 1 -- 172.21.15.136:0/1460644334 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f411c0f53b0 con 0x7f41200ca7d0 2024-04-25T11:25:43.355 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:43.355 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 msgr2=0x7f4104071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 0x7f4104071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4120151160 tx=0x7f4110008040 comp rx=0 tx=0).stop 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 msgr2=0x7f41201f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f411c0ae040 tx=0x7f411c0192c0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 shutdown_connections 2024-04-25T11:25:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41200b0530 0x7f4120148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f410406f400 0x7f4104071910 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41200cf9d0 0x7f4120150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 --2- 172.21.15.136:0/1460644334 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41200ca7d0 0x7f41201f9050 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 >> 172.21.15.136:0/1460644334 conn(0x7f4120070f70 msgr2=0x7f412006f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 shutdown_connections 2024-04-25T11:25:43.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:43.362+0000 7f4127b0c640 1 -- 172.21.15.136:0/1460644334 wait complete. 2024-04-25T11:25:43.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:45.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:45.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 Processor -- start 2024-04-25T11:25:45.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 -- start start 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2e540ce260 0x7f2e540ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540ca010 0x7f2e540ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e540cbad0 con 0x7f2e540c8cd0 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2e540c8550 con 0x7f2e540ce260 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e5b624640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e540c7c20 con 0x7f2e540ca010 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2e540ce260 0x7f2e540ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2e540ce260 0x7f2e540ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51986/0 (socket says 172.21.15.136:51986) 2024-04-25T11:25:45.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3243515662 learned_addr learned my addr 172.21.15.136:0/3243515662 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e593c0640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2e540c8cd0 0x7f2e5414f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57702/0 (socket says 172.21.15.136:57702) 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e59bc1640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540ca010 0x7f2e540ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3243515662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540ca010 msgr2=0x7f2e540ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540ca010 0x7f2e540ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3243515662 >> v1:172.21.15.136:6789/0 conn(0x7f2e540c8cd0 legacy=0x7f2e5414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3243515662 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e540c7f00 con 0x7f2e540ce260 2024-04-25T11:25:45.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.534+0000 7f2e4bfff640 1 -- 172.21.15.136:0/3243515662 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 517593936 0 0) 0x7f2e540cbad0 con 0x7f2e540c8cd0 2024-04-25T11:25:45.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e58bbf640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ce260 0x7f2e540ce670 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2e540c86d0 tx=0x7f2e44021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301319572b7d8cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:45.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e4bfff640 1 -- 172.21.15.136:0/3243515662 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440b6070 con 0x7f2e540ce260 2024-04-25T11:25:45.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e4bfff640 1 -- 172.21.15.136:0/3243515662 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e440b1600 con 0x7f2e540ce260 2024-04-25T11:25:45.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e4bfff640 1 -- 172.21.15.136:0/3243515662 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440b19d0 con 0x7f2e540ce260 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 -- 172.21.15.136:0/3243515662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ce260 msgr2=0x7f2e540ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ce260 0x7f2e540ce670 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2e540c86d0 tx=0x7f2e44021400 comp rx=0 tx=0).stop 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 -- 172.21.15.136:0/3243515662 shutdown_connections 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540ca010 0x7f2e540ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3243515662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ce260 0x7f2e540ce670 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 -- 172.21.15.136:0/3243515662 >> 172.21.15.136:0/3243515662 conn(0x7f2e5406f9d0 msgr2=0x7f2e5406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:45.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 -- 172.21.15.136:0/3243515662 shutdown_connections 2024-04-25T11:25:45.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 -- 172.21.15.136:0/3243515662 wait complete. 2024-04-25T11:25:45.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.538+0000 7f2e5b624640 1 Processor -- start 2024-04-25T11:25:45.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- start start 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 0x7f2e541543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54147550 0x7f2e54147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540cb720 con 0x7f2e540ca010 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e540ca960 con 0x7f2e54147550 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e54154ce0 con 0x7f2e540c8cd0 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51990/0 (socket says 172.21.15.136:51990) 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3212778686 learned_addr learned my addr 172.21.15.136:0/3212778686 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e593c0640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 0x7f2e541543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e59bc1640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54147550 0x7f2e54147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 msgr2=0x7f2e541543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 0x7f2e541543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54147550 msgr2=0x7f2e54147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54147550 0x7f2e54147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 -- 172.21.15.136:0/3212778686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5414daf0 con 0x7f2e540ca010 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e593c0640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 0x7f2e541543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e58bbf640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2e440b4040 tx=0x7f2e4401fc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440c1070 con 0x7f2e540ca010 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5414ddd0 con 0x7f2e540ca010 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e54154a20 con 0x7f2e540ca010 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e44004030 con 0x7f2e540ca010 2024-04-25T11:25:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440b6040 con 0x7f2e540ca010 2024-04-25T11:25:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e340052a0 con 0x7f2e540ca010 2024-04-25T11:25:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.542+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2e440c4070 con 0x7f2e540ca010 2024-04-25T11:25:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.546+0000 7f2e49ffb640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 0x7f2e38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.546+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f2e4412d1b0 con 0x7f2e540ca010 2024-04-25T11:25:45.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.546+0000 7f2e593c0640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 0x7f2e38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:45.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.546+0000 7f2e593c0640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 0x7f2e38071910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e50007fd0 tx=0x7f2e50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:45.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.550+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2e4401af60 con 0x7f2e540ca010 2024-04-25T11:25:45.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.778+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e34005860 con 0x7f2e540ca010 2024-04-25T11:25:45.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.778+0000 7f2e49ffb640 1 -- 172.21.15.136:0/3212778686 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f2e440f9570 con 0x7f2e540ca010 2024-04-25T11:25:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:45.775 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:45.775 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:45.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 msgr2=0x7f2e38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 0x7f2e38071910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e50007fd0 tx=0x7f2e50009040 comp rx=0 tx=0).stop 2024-04-25T11:25:45.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 msgr2=0x7f2e54146c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:45.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2e440b4040 tx=0x7f2e4401fc40 comp rx=0 tx=0).stop 2024-04-25T11:25:45.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 shutdown_connections 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e54147550 0x7f2e54147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e540ca010 0x7f2e54146c60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e3806f400 0x7f2e38071910 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 --2- 172.21.15.136:0/3212778686 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e540c8cd0 0x7f2e541543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 >> 172.21.15.136:0/3212778686 conn(0x7f2e5406f9d0 msgr2=0x7f2e54158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 shutdown_connections 2024-04-25T11:25:45.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:45.782+0000 7f2e5b624640 1 -- 172.21.15.136:0/3212778686 wait complete. 2024-04-25T11:25:45.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:47.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:47.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.970+0000 7f275b5d3640 1 Processor -- start 2024-04-25T11:25:47.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.970+0000 7f275b5d3640 1 -- start start 2024-04-25T11:25:47.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275b5d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f27540b5060 0x7f27540c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275b5d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27540ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275b5d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f275415d750 con 0x7f275415a430 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275b5d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f275415da30 con 0x7f27540b5060 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275b5d3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f275415dd10 con 0x7f27540ca290 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f27540b5060 0x7f27540c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f27540b5060 0x7f27540c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52002/0 (socket says 172.21.15.136:52002) 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 -- 172.21.15.136:0/4033967264 learned_addr learned my addr 172.21.15.136:0/4033967264 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f2758b6e640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27540ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 -- 172.21.15.136:0/4033967264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 msgr2=0x7f27540ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27540ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 -- 172.21.15.136:0/4033967264 >> v1:172.21.15.136:6789/0 conn(0x7f275415a430 legacy=0x7f275415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.974+0000 7f275936f640 1 -- 172.21.15.136:0/4033967264 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275415dff0 con 0x7f27540b5060 2024-04-25T11:25:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275936f640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27540b5060 0x7f27540c4520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2750009870 tx=0x7f275001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22dd13beb502d7a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:47.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f274bfff640 1 -- 172.21.15.136:0/4033967264 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500b5070 con 0x7f27540b5060 2024-04-25T11:25:47.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f274bfff640 1 -- 172.21.15.136:0/4033967264 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27500a8a50 con 0x7f27540b5060 2024-04-25T11:25:47.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f274bfff640 1 -- 172.21.15.136:0/4033967264 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500a8e20 con 0x7f27540b5060 2024-04-25T11:25:47.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- 172.21.15.136:0/4033967264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27540b5060 msgr2=0x7f27540c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27540b5060 0x7f27540c4520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2750009870 tx=0x7f275001ed50 comp rx=0 tx=0).stop 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- 172.21.15.136:0/4033967264 shutdown_connections 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27540ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 --2- 172.21.15.136:0/4033967264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27540b5060 0x7f27540c4520 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- 172.21.15.136:0/4033967264 >> 172.21.15.136:0/4033967264 conn(0x7f2754070f70 msgr2=0x7f2754068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- 172.21.15.136:0/4033967264 shutdown_connections 2024-04-25T11:25:47.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- 172.21.15.136:0/4033967264 wait complete. 2024-04-25T11:25:47.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 Processor -- start 2024-04-25T11:25:47.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.978+0000 7f275b5d3640 1 -- start start 2024-04-25T11:25:47.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27540b5060 0x7f27540b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27541f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f275415d3f0 con 0x7f275415a430 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2754159f10 con 0x7f27540b5060 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27540c4a60 con 0x7f27540ca290 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52016/0 (socket says 172.21.15.136:52016) 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 -- 172.21.15.136:0/2564297521 learned_addr learned my addr 172.21.15.136:0/2564297521 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2758b6e640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27541f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275936f640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27540b5060 0x7f27540b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 -- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 msgr2=0x7f27541f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27541f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 -- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27540b5060 msgr2=0x7f27540b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:47.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27540b5060 0x7f27540b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 -- 172.21.15.136:0/2564297521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27541f30a0 con 0x7f275415a430 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2759b70640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f274c00c760 tx=0x7f274c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c003b80 con 0x7f275415a430 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c003d20 con 0x7f275415a430 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c00aa30 con 0x7f275415a430 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275406ec00 con 0x7f275415a430 2024-04-25T11:25:47.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275406f1c0 con 0x7f275415a430 2024-04-25T11:25:47.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.982+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f275406a350 con 0x7f275415a430 2024-04-25T11:25:47.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.986+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f274c00abd0 con 0x7f275415a430 2024-04-25T11:25:47.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.986+0000 7f2749ffb640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 0x7f2738071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:47.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.986+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f274c08f560 con 0x7f275415a430 2024-04-25T11:25:47.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.986+0000 7f275936f640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 0x7f2738071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:47.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.986+0000 7f275936f640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 0x7f2738071910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f27500b2000 tx=0x7f27500aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:47.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:47.990+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f274c05c080 con 0x7f275415a430 2024-04-25T11:25:48.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.218+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27540cadd0 con 0x7f275415a430 2024-04-25T11:25:48.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f2749ffb640 1 -- 172.21.15.136:0/2564297521 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f274c064070 con 0x7f275415a430 2024-04-25T11:25:48.216 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:48.216 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:48.216 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:48.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 msgr2=0x7f2738071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:48.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 0x7f2738071910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f27500b2000 tx=0x7f27500aa070 comp rx=0 tx=0).stop 2024-04-25T11:25:48.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 msgr2=0x7f27541e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:48.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f274c00c760 tx=0x7f274c00cc20 comp rx=0 tx=0).stop 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 shutdown_connections 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f275415a430 0x7f27541e7780 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f273806f400 0x7f2738071910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27540ca290 0x7f27541f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 --2- 172.21.15.136:0/2564297521 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27540b5060 0x7f27540b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 >> 172.21.15.136:0/2564297521 conn(0x7f2754070f70 msgr2=0x7f2754153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 shutdown_connections 2024-04-25T11:25:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:48.222+0000 7f275b5d3640 1 -- 172.21.15.136:0/2564297521 wait complete. 2024-04-25T11:25:48.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:50.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:50.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 Processor -- start 2024-04-25T11:25:50.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 -- start start 2024-04-25T11:25:50.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c96e0 0x7fa3181540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3180c79e0 0x7fa3180c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3180c9af0 con 0x7fa31815a3d0 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa31806cc10 con 0x7fa3180c79e0 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31fa53640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3180cb220 con 0x7fa3180c96e0 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31cfee640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3180c79e0 0x7fa3180c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31cfee640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3180c79e0 0x7fa3180c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52018/0 (socket says 172.21.15.136:52018) 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.430+0000 7fa31cfee640 1 -- 172.21.15.136:0/1831108646 learned_addr learned my addr 172.21.15.136:0/1831108646 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31d7ef640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c96e0 0x7fa3181540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31cfee640 1 -- 172.21.15.136:0/1831108646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c96e0 msgr2=0x7fa3181540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31cfee640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c96e0 0x7fa3181540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31cfee640 1 -- 172.21.15.136:0/1831108646 >> v1:172.21.15.157:6789/0 conn(0x7fa31815a3d0 legacy=0x7fa31815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:50.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31cfee640 1 -- 172.21.15.136:0/1831108646 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3180cb500 con 0x7fa3180c79e0 2024-04-25T11:25:50.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31cfee640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3180c79e0 0x7fa3180c7df0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa308009870 tx=0x7fa30801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4f14bc7c7a2a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:50.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa30ffff640 1 -- 172.21.15.136:0/1831108646 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3080b5070 con 0x7fa3180c79e0 2024-04-25T11:25:50.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa30ffff640 1 -- 172.21.15.136:0/1831108646 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3080a89d0 con 0x7fa3180c79e0 2024-04-25T11:25:50.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa30ffff640 1 -- 172.21.15.136:0/1831108646 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3080a8da0 con 0x7fa3180c79e0 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 -- 172.21.15.136:0/1831108646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3180c79e0 msgr2=0x7fa3180c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3180c79e0 0x7fa3180c7df0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa308009870 tx=0x7fa30801ed10 comp rx=0 tx=0).stop 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 -- 172.21.15.136:0/1831108646 shutdown_connections 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3180c79e0 0x7fa3180c7df0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1831108646 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c96e0 0x7fa3181540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 -- 172.21.15.136:0/1831108646 >> 172.21.15.136:0/1831108646 conn(0x7fa31806f9d0 msgr2=0x7fa31806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 -- 172.21.15.136:0/1831108646 shutdown_connections 2024-04-25T11:25:50.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.434+0000 7fa31fa53640 1 -- 172.21.15.136:0/1831108646 wait complete. 2024-04-25T11:25:50.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 Processor -- start 2024-04-25T11:25:50.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 -- start start 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c79e0 0x7fa3181f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 0x7fa318159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa3180cf0f0 con 0x7fa31815a3d0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.438+0000 7fa31fa53640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa318154610 con 0x7fa3180c96e0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31fa53640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3180ced40 con 0x7fa3180c79e0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52030/0 (socket says 172.21.15.136:52030) 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 -- 172.21.15.136:0/1839247877 learned_addr learned my addr 172.21.15.136:0/1839247877 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31cfee640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 0x7fa318159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31d7ef640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c79e0 0x7fa3181f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 -- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c79e0 msgr2=0x7fa3181f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c79e0 0x7fa3181f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 -- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 msgr2=0x7fa318159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 0x7fa318159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 -- 172.21.15.136:0/1839247877 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3181f5dd0 con 0x7fa31815a3d0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31cfee640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 0x7fa318159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31dff0640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b440 tx=0x7fa31000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310015020 con 0x7fa31815a3d0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa310011870 con 0x7fa31815a3d0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3181f6040 con 0x7fa31815a3d0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310011c60 con 0x7fa31815a3d0 2024-04-25T11:25:50.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3181f6b20 con 0x7fa31815a3d0 2024-04-25T11:25:50.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa318158dd0 con 0x7fa31815a3d0 2024-04-25T11:25:50.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa310018020 con 0x7fa31815a3d0 2024-04-25T11:25:50.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 0x7fa2fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:50.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fa31008f1e0 con 0x7fa31815a3d0 2024-04-25T11:25:50.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.442+0000 7fa31d7ef640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 0x7fa2fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:50.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.446+0000 7fa31d7ef640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 0x7fa2fc071940 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa314003fd0 tx=0x7fa3140039a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:50.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.450+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa31001f030 con 0x7fa31815a3d0 2024-04-25T11:25:50.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:25:50.663 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:25:50.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.682+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa31806a350 con 0x7fa31815a3d0 2024-04-25T11:25:50.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.682+0000 7fa30dffb640 1 -- 172.21.15.136:0/1839247877 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7fa31005bc70 con 0x7fa31815a3d0 2024-04-25T11:25:50.678 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:50.678 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:50.678 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 msgr2=0x7fa2fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 0x7fa2fc071940 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa314003fd0 tx=0x7fa3140039a0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 msgr2=0x7fa31814f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b440 tx=0x7fa31000b900 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 shutdown_connections 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa31815a3d0 0x7fa31814f7d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3180c96e0 0x7fa318159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa2fc06f430 0x7fa2fc071940 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 --2- 172.21.15.136:0/1839247877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3180c79e0 0x7fa3181f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 >> 172.21.15.136:0/1839247877 conn(0x7fa31806f9d0 msgr2=0x7fa3180ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 shutdown_connections 2024-04-25T11:25:50.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:50.686+0000 7fa31fa53640 1 -- 172.21.15.136:0/1839247877 wait complete. 2024-04-25T11:25:50.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:52.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:52.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.882+0000 7f350e18a640 1 Processor -- start 2024-04-25T11:25:52.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.882+0000 7f350e18a640 1 -- start start 2024-04-25T11:25:52.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f350e18a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f35080ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f350e18a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35080cbad0 con 0x7f35080ca010 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f350e18a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35080c7c20 con 0x7f35080c8cd0 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f350e18a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35080c8550 con 0x7f35080ce260 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3507fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f35080ca010 0x7f35080ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57718/0 (socket says 172.21.15.136:57718) 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3507fff640 1 -- 172.21.15.136:0/2157688534 learned_addr learned my addr 172.21.15.136:0/2157688534 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f35080ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889972095 0 0) 0x7f35080cbad0 con 0x7f35080ca010 2024-04-25T11:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34f0003530 con 0x7f35080ca010 2024-04-25T11:25:52.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4118752015 0 0) 0x7f34f0003530 con 0x7f35080ca010 2024-04-25T11:25:52.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 msgr2=0x7f35080ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:52.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f35080ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:52.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 >> v1:172.21.15.157:6789/0 conn(0x7f35080c8cd0 legacy=0x7f350814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35080c7f00 con 0x7f35080ca010 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f3500001a00 con 0x7f35080ca010 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 >> v1:172.21.15.136:6789/0 conn(0x7f35080ca010 legacy=0x7f35080ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34f00039f0 0x7f34f0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f34f0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f34f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35080c8550 con 0x7f34f0004810 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34f000bb90 con 0x7f34f00039f0 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34f000dfa0 con 0x7f35080ce260 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f34f0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f35077fe640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34f00039f0 0x7f34f0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3507fff640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f34f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 -- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 msgr2=0x7f34f0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:52.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080ce260 0x7f34f0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:52.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 -- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34f00039f0 msgr2=0x7f34f0003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:52.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34f00039f0 0x7f34f0003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:52.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.886+0000 7f3506ffd640 1 -- 172.21.15.136:0/2157688534 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f0003530 con 0x7f34f0004810 2024-04-25T11:25:52.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f3506ffd640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f34f0004c20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ab0 tx=0x7f34fc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2dad2b42d160a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:52.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc0b2070 con 0x7f34f0004810 2024-04-25T11:25:52.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc0a5660 con 0x7f34f0004810 2024-04-25T11:25:52.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f35067fc640 1 -- 172.21.15.136:0/2157688534 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc0096b0 con 0x7f34f0004810 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 -- 172.21.15.136:0/2157688534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 msgr2=0x7f34f0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 --2- 172.21.15.136:0/2157688534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f34f0004c20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009ab0 tx=0x7f34fc01ca70 comp rx=0 tx=0).stop 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f3507fff640 1 -- 172.21.15.136:0/2157688534 reap_dead start 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 -- 172.21.15.136:0/2157688534 shutdown_connections 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 -- 172.21.15.136:0/2157688534 >> 172.21.15.136:0/2157688534 conn(0x7f350806f9d0 msgr2=0x7f350806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 -- 172.21.15.136:0/2157688534 shutdown_connections 2024-04-25T11:25:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.890+0000 7f350e18a640 1 -- 172.21.15.136:0/2157688534 wait complete. 2024-04-25T11:25:52.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 Processor -- start 2024-04-25T11:25:52.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- start start 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080cd800 0x7f3508142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35080cdc10 0x7f3508142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35080cb720 con 0x7f34f0004810 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f35080c91e0 con 0x7f35080cdc10 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3508147200 con 0x7f35080cd800 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52052/0 (socket says 172.21.15.136:52052) 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 -- 172.21.15.136:0/1278867473 learned_addr learned my addr 172.21.15.136:0/1278867473 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:52.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f3507fff640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35080cdc10 0x7f3508142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f3506ffd640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080cd800 0x7f3508142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 -- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080cd800 msgr2=0x7f3508142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080cd800 0x7f3508142350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 -- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35080cdc10 msgr2=0x7f3508142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35080cdc10 0x7f3508142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 -- 172.21.15.136:0/1278867473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3508146d30 con 0x7f34f0004810 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f35077fe640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f34f800e580 tx=0x7f34f800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f80052e0 con 0x7f34f0004810 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f8018070 con 0x7f34f0004810 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3508148370 con 0x7f34f0004810 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8003820 con 0x7f34f0004810 2024-04-25T11:25:52.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.894+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350814f7b0 con 0x7f34f0004810 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350814fe70 con 0x7f34f0004810 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f34f80039c0 con 0x7f34f0004810 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f34effff640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 0x7f34f4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f34f808f410 con 0x7f34f0004810 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f3506ffd640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 0x7f34f4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.898+0000 7f3506ffd640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 0x7f34f4071940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0af000 tx=0x7f34fc0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:52.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:52.902+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f350814fe70 con 0x7f34f0004810 2024-04-25T11:25:53.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.134+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f350806a350 con 0x7f34f0004810 2024-04-25T11:25:53.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.134+0000 7f34effff640 1 -- 172.21.15.136:0/1278867473 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f34f805bf40 con 0x7f34f0004810 2024-04-25T11:25:53.131 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:53.131 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 msgr2=0x7f34f4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 0x7f34f4071940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0af000 tx=0x7f34fc0046d0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 msgr2=0x7f35081467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f34f800e580 tx=0x7f34f800ea40 comp rx=0 tx=0).stop 2024-04-25T11:25:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 shutdown_connections 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35080cdc10 0x7f3508142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34f406f430 0x7f34f4071940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35080cd800 0x7f3508142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 --2- 172.21.15.136:0/1278867473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34f0004810 0x7f35081467f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 >> 172.21.15.136:0/1278867473 conn(0x7f350806f9d0 msgr2=0x7f3508148f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 shutdown_connections 2024-04-25T11:25:53.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:53.138+0000 7f350e18a640 1 -- 172.21.15.136:0/1278867473 wait complete. 2024-04-25T11:25:53.155 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:55.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:55.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 Processor -- start 2024-04-25T11:25:55.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 -- start start 2024-04-25T11:25:55.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3464156750 0x7f34640ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 0x7f34640cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f346415c410 con 0x7f34640cf9d0 2024-04-25T11:25:55.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f34640cfde0 con 0x7f3464156750 2024-04-25T11:25:55.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f3468a85640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f346415c6b0 con 0x7f34640ca7d0 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3464156750 0x7f34640ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3464156750 0x7f34640ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40048/0 (socket says 172.21.15.136:40048) 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 -- 172.21.15.136:0/836215990 learned_addr learned my addr 172.21.15.136:0/836215990 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f345bfff640 1 --2- 172.21.15.136:0/836215990 >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 0x7f34640cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 -- 172.21.15.136:0/836215990 >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 msgr2=0x7f34640cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 --2- 172.21.15.136:0/836215990 >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 0x7f34640cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 -- 172.21.15.136:0/836215990 >> v1:172.21.15.157:6789/0 conn(0x7f34640cf9d0 legacy=0x7f346406e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f346259c640 1 -- 172.21.15.136:0/836215990 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346415c990 con 0x7f3464156750 2024-04-25T11:25:55.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.326+0000 7f345bfff640 1 --2- 172.21.15.136:0/836215990 >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 0x7f34640cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:25:55.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f346259c640 1 --2- 172.21.15.136:0/836215990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3464156750 0x7f34640ca290 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f346406f960 tx=0x7f345401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bae0dd3318beaf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3461d9b640 1 -- 172.21.15.136:0/836215990 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540b0070 con 0x7f3464156750 2024-04-25T11:25:55.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3461d9b640 1 -- 172.21.15.136:0/836215990 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34540ab600 con 0x7f3464156750 2024-04-25T11:25:55.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3461d9b640 1 -- 172.21.15.136:0/836215990 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540ab9d0 con 0x7f3464156750 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 -- 172.21.15.136:0/836215990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3464156750 msgr2=0x7f34640ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 --2- 172.21.15.136:0/836215990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3464156750 0x7f34640ca290 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f346406f960 tx=0x7f345401c4e0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 -- 172.21.15.136:0/836215990 shutdown_connections 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 --2- 172.21.15.136:0/836215990 >> v2:172.21.15.157:3300/0 conn(0x7f34640ca7d0 0x7f34640cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 --2- 172.21.15.136:0/836215990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3464156750 0x7f34640ca290 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 -- 172.21.15.136:0/836215990 >> 172.21.15.136:0/836215990 conn(0x7f3464070f70 msgr2=0x7f3464068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 -- 172.21.15.136:0/836215990 shutdown_connections 2024-04-25T11:25:55.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.330+0000 7f3468a85640 1 -- 172.21.15.136:0/836215990 wait complete. 2024-04-25T11:25:55.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 Processor -- start 2024-04-25T11:25:55.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- start start 2024-04-25T11:25:55.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34640ca7d0 0x7f34641f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34640b05a0 0x7f346414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3464169440 con 0x7f34640cf9d0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34641692c0 con 0x7f34640b05a0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34641695c0 con 0x7f34640ca7d0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40054/0 (socket says 172.21.15.136:40054) 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 -- 172.21.15.136:0/1702918811 learned_addr learned my addr 172.21.15.136:0/1702918811 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f346259c640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34640ca7d0 0x7f34641f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3462d9d640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34640b05a0 0x7f346414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 -- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34640ca7d0 msgr2=0x7f34641f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34640ca7d0 0x7f34641f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 -- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34640b05a0 msgr2=0x7f346414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34640b05a0 0x7f346414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 -- 172.21.15.136:0/1702918811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34640b3ba0 con 0x7f34640cf9d0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345bfff640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c820 tx=0x7f344c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c015070 con 0x7f34640cf9d0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344c007a30 con 0x7f34640cf9d0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c007de0 con 0x7f34640cf9d0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3464153360 con 0x7f34640cf9d0 2024-04-25T11:25:55.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.334+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f346415b580 con 0x7f34640cf9d0 2024-04-25T11:25:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f346406a350 con 0x7f34640cf9d0 2024-04-25T11:25:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f344c018070 con 0x7f34640cf9d0 2024-04-25T11:25:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f345affd640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 0x7f3444071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f344c08f6b0 con 0x7f34640cf9d0 2024-04-25T11:25:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f346259c640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 0x7f3444071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:55.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.338+0000 7f346259c640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 0x7f3444071910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34540ad000 tx=0x7f3454008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:55.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.346+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f344c05c140 con 0x7f34640cf9d0 2024-04-25T11:25:55.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.638+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34640cbf50 con 0x7f34640cf9d0 2024-04-25T11:25:55.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.638+0000 7f345affd640 1 -- 172.21.15.136:0/1702918811 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f344c05ba40 con 0x7f34640cf9d0 2024-04-25T11:25:55.635 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:55.635 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:55.635 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:55.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 msgr2=0x7f3444071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 0x7f3444071910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f34540ad000 tx=0x7f3454008040 comp rx=0 tx=0).stop 2024-04-25T11:25:55.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 msgr2=0x7f34640b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:55.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c820 tx=0x7f344c00cce0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 shutdown_connections 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34640b05a0 0x7f346414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34640cf9d0 0x7f34640b4590 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f344406f400 0x7f3444071910 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 --2- 172.21.15.136:0/1702918811 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34640ca7d0 0x7f34641f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 >> 172.21.15.136:0/1702918811 conn(0x7f3464070f70 msgr2=0x7f3464068620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 shutdown_connections 2024-04-25T11:25:55.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:55.642+0000 7f3468a85640 1 -- 172.21.15.136:0/1702918811 wait complete. 2024-04-25T11:25:55.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:25:57.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:25:57.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.822+0000 7f184b890640 1 Processor -- start 2024-04-25T11:25:57.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.822+0000 7f184b890640 1 -- start start 2024-04-25T11:25:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184b890640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f184414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184b890640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18440cbad0 con 0x7f18440ca720 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184b890640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18440c7c20 con 0x7f18440ce260 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184b890640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f18440c8550 con 0x7f18440c8cd0 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1849e2d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f18440ca720 0x7f18441637f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50326/0 (socket says 172.21.15.136:50326) 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1849e2d640 1 -- 172.21.15.136:0/3156625312 learned_addr learned my addr 172.21.15.136:0/3156625312 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184962c640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f184414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471370020 0 0) 0x7f18440cbad0 con 0x7f18440ca720 2024-04-25T11:25:57.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1830003530 con 0x7f18440ca720 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1251906055 0 0) 0x7f1830003530 con 0x7f18440ca720 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 msgr2=0x7f184414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f184414f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 >> v1:172.21.15.157:6789/0 conn(0x7f18440ce260 legacy=0x7f18440ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18440c7f00 con 0x7f18440ca720 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f183c001a00 con 0x7f18440ca720 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 >> v1:172.21.15.136:6789/0 conn(0x7f18440ca720 legacy=0x7f18441637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f18300039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f1830004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1830004d50 0x7f1830005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18440c8550 con 0x7f1830003f70 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f183000de00 con 0x7f1830004d50 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f183000bbf0 con 0x7f18440c8cd0 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f1830004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f184962c640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f18300039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1849e2d640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1830004d50 0x7f1830005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 -- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 msgr2=0x7f18300039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440c8cd0 0x7f18300039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 -- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1830004d50 msgr2=0x7f1830005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1830004d50 0x7f1830005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 -- 172.21.15.136:0/3156625312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1830003530 con 0x7f1830003f70 2024-04-25T11:25:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f1830004810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1834001780 tx=0x7f183401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21ec2883a72d8db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18340013a0 con 0x7f1830003f70 2024-04-25T11:25:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18340a9070 con 0x7f1830003f70 2024-04-25T11:25:57.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.826+0000 7f183bfff640 1 -- 172.21.15.136:0/3156625312 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834003600 con 0x7f1830003f70 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- 172.21.15.136:0/3156625312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 msgr2=0x7f1830004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 --2- 172.21.15.136:0/3156625312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f1830004810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1834001780 tx=0x7f183401c420 comp rx=0 tx=0).stop 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f1849e2d640 1 -- 172.21.15.136:0/3156625312 reap_dead start 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- 172.21.15.136:0/3156625312 shutdown_connections 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- 172.21.15.136:0/3156625312 >> 172.21.15.136:0/3156625312 conn(0x7f184406f9d0 msgr2=0x7f184406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- 172.21.15.136:0/3156625312 shutdown_connections 2024-04-25T11:25:57.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- 172.21.15.136:0/3156625312 wait complete. 2024-04-25T11:25:57.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 Processor -- start 2024-04-25T11:25:57.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.830+0000 7f184b890640 1 -- start start 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 0x7f184415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1844153760 0x7f1844148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18440cb720 con 0x7f1830003f70 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18440c91e0 con 0x7f1844153760 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f184415bcd0 con 0x7f184415a600 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40068/0 (socket says 172.21.15.136:40068) 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 -- 172.21.15.136:0/3936492427 learned_addr learned my addr 172.21.15.136:0/3936492427 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:25:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 0x7f184415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1849e2d640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1844153760 0x7f1844148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 -- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 msgr2=0x7f184415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 0x7f184415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 -- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1844153760 msgr2=0x7f1844148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1844153760 0x7f1844148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 -- 172.21.15.136:0/3936492427 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184415b800 con 0x7f1830003f70 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 0x7f184415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184962c640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f18400115e0 tx=0x7f1840011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184000f8f0 con 0x7f1830003f70 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184000fa90 con 0x7f1830003f70 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18440cd3d0 con 0x7f1830003f70 2024-04-25T11:25:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184000fe40 con 0x7f1830003f70 2024-04-25T11:25:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18441f1fa0 con 0x7f1830003f70 2024-04-25T11:25:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1844154cb0 con 0x7f1830003f70 2024-04-25T11:25:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1840015460 con 0x7f1830003f70 2024-04-25T11:25:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 0x7f1828071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:25:57.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f184008ffe0 con 0x7f1830003f70 2024-04-25T11:25:57.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.834+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 0x7f1828071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:25:57.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.838+0000 7f1848e2b640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 0x7f1828071910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f18340ad000 tx=0x7f18340011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:25:57.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:57.842+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1844154cb0 con 0x7f1830003f70 2024-04-25T11:25:58.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.050+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f184406a350 con 0x7f1830003f70 2024-04-25T11:25:58.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.050+0000 7f1839ffb640 1 -- 172.21.15.136:0/3936492427 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6926 (secure 0 0 0) 0x7f1840019030 con 0x7f1830003f70 2024-04-25T11:25:58.047 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:25:58.047 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5238,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:23:44.133427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"a","rank":4,"incarnation":20,"state":"up:active","state_seq":57,"addr":"172.21.15.136:6833/3353376405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3353376405},{"type":"v1","addr":"172.21.15.136:6833","nonce":3353376405}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260,4254]},"id":1}]} 2024-04-25T11:25:58.047 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 25 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 msgr2=0x7f1828071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 0x7f1828071910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f18340ad000 tx=0x7f18340011a0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 msgr2=0x7f18440cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f18400115e0 tx=0x7f1840011aa0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 shutdown_connections 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1844153760 0x7f1844148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f182806f400 0x7f1828071910 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f184415a600 0x7f184415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 --2- 172.21.15.136:0/3936492427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1830003f70 0x7f18440cd7e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:25:58.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 >> 172.21.15.136:0/3936492427 conn(0x7f184406f9d0 msgr2=0x7f184414cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:25:58.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 shutdown_connections 2024-04-25T11:25:58.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:25:58.054+0000 7f184b890640 1 -- 172.21.15.136:0/3936492427 wait complete. 2024-04-25T11:25:58.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:26:00.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:00.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.230+0000 7f9144d88640 1 Processor -- start 2024-04-25T11:26:00.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.230+0000 7f9144d88640 1 -- start start 2024-04-25T11:26:00.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9144d88640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9140156330 0x7f914015cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9144d88640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f91400cdee0 0x7f91400ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9144d88640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914015d010 con 0x7f91400b5490 2024-04-25T11:26:00.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9144d88640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f91401602f0 con 0x7f9140156330 2024-04-25T11:26:00.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9144d88640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f91400ccb80 con 0x7f91400cdee0 2024-04-25T11:26:00.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9140156330 0x7f914015cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913ed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f91400b5490 0x7f91400b58a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50338/0 (socket says 172.21.15.136:50338) 2024-04-25T11:26:00.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9140156330 0x7f914015cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40084/0 (socket says 172.21.15.136:40084) 2024-04-25T11:26:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913ed9d640 1 -- 172.21.15.136:0/260125937 learned_addr learned my addr 172.21.15.136:0/260125937 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:26:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f9137fff640 1 --2- 172.21.15.136:0/260125937 >> v2:172.21.15.157:3300/0 conn(0x7f91400cdee0 0x7f91400ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 -- 172.21.15.136:0/260125937 >> v2:172.21.15.157:3300/0 conn(0x7f91400cdee0 msgr2=0x7f91400ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 --2- 172.21.15.136:0/260125937 >> v2:172.21.15.157:3300/0 conn(0x7f91400cdee0 0x7f91400ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 -- 172.21.15.136:0/260125937 >> v1:172.21.15.136:6789/0 conn(0x7f91400b5490 legacy=0x7f91400b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913dd9b640 1 -- 172.21.15.136:0/260125937 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2879546254 0 0) 0x7f91380014a0 con 0x7f91400b5490 2024-04-25T11:26:00.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 -- 172.21.15.136:0/260125937 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9140156740 con 0x7f9140156330 2024-04-25T11:26:00.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913e59c640 1 --2- 172.21.15.136:0/260125937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9140156330 0x7f914015cad0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f91400b49a0 tx=0x7f913001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac40edf1048c691 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913dd9b640 1 -- 172.21.15.136:0/260125937 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91300b5070 con 0x7f9140156330 2024-04-25T11:26:00.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913dd9b640 1 -- 172.21.15.136:0/260125937 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91300a8970 con 0x7f9140156330 2024-04-25T11:26:00.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.234+0000 7f913dd9b640 1 -- 172.21.15.136:0/260125937 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91300a8d40 con 0x7f9140156330 2024-04-25T11:26:00.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- 172.21.15.136:0/260125937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9140156330 msgr2=0x7f914015cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- 172.21.15.136:0/260125937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9140156330 0x7f914015cad0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f91400b49a0 tx=0x7f913001ec70 comp rx=0 tx=0).stop 2024-04-25T11:26:00.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- 172.21.15.136:0/260125937 shutdown_connections 2024-04-25T11:26:00.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- 172.21.15.136:0/260125937 >> v2:172.21.15.157:3300/0 conn(0x7f91400cdee0 0x7f91400ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- 172.21.15.136:0/260125937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9140156330 0x7f914015cad0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- 172.21.15.136:0/260125937 >> 172.21.15.136:0/260125937 conn(0x7f914006d6c0 msgr2=0x7f914006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:00.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- 172.21.15.136:0/260125937 shutdown_connections 2024-04-25T11:26:00.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- 172.21.15.136:0/260125937 wait complete. 2024-04-25T11:26:00.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 Processor -- start 2024-04-25T11:26:00.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- start start 2024-04-25T11:26:00.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91400b5490 0x7f91400d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9140156330 0x7f9140155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f914015fca0 con 0x7f91400cdee0 2024-04-25T11:26:00.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f914015f9c0 con 0x7f91400b5490 2024-04-25T11:26:00.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9144d88640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f914015ff80 con 0x7f9140156330 2024-04-25T11:26:00.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9137fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9137fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40100/0 (socket says 172.21.15.136:40100) 2024-04-25T11:26:00.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f9137fff640 1 -- 172.21.15.136:0/1465796303 learned_addr learned my addr 172.21.15.136:0/1465796303 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:00.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f913e59c640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91400b5490 0x7f91400d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.238+0000 7f913ed9d640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9140156330 0x7f9140155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 -- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9140156330 msgr2=0x7f9140155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9140156330 0x7f9140155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 -- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91400b5490 msgr2=0x7f91400d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91400b5490 0x7f91400d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 -- 172.21.15.136:0/1465796303 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91400d1630 con 0x7f91400cdee0 2024-04-25T11:26:00.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9137fff640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f912800d7d0 tx=0x7f912800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912800b6c0 con 0x7f91400cdee0 2024-04-25T11:26:00.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128019070 con 0x7f91400cdee0 2024-04-25T11:26:00.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91401544f0 con 0x7f91400cdee0 2024-04-25T11:26:00.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912800ba80 con 0x7f91400cdee0 2024-04-25T11:26:00.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914015f000 con 0x7f91400cdee0 2024-04-25T11:26:00.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9140154260 con 0x7f91400cdee0 2024-04-25T11:26:00.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f912800bc20 con 0x7f91400cdee0 2024-04-25T11:26:00.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 0x7f9120071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f912808fa70 con 0x7f91400cdee0 2024-04-25T11:26:00.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.242+0000 7f913e59c640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 0x7f9120071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.246+0000 7f913e59c640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 0x7f9120071940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f91300b2000 tx=0x7f91300aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.250+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f912805c5d0 con 0x7f91400cdee0 2024-04-25T11:26:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.498+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f914006a350 con 0x7f91400cdee0 2024-04-25T11:26:00.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.498+0000 7f9136ffd640 1 -- 172.21.15.136:0/1465796303 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6093 (secure 0 0 0) 0x7f912805bed0 con 0x7f91400cdee0 2024-04-25T11:26:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:00.495 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:25:58.820536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:26:00.496 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 26 2024-04-25T11:26:00.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 msgr2=0x7f9120071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 0x7f9120071940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f91300b2000 tx=0x7f91300aa070 comp rx=0 tx=0).stop 2024-04-25T11:26:00.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 msgr2=0x7f91400cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f912800d7d0 tx=0x7f912800dc90 comp rx=0 tx=0).stop 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 shutdown_connections 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f912006f430 0x7f9120071940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9140156330 0x7f9140155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91400cdee0 0x7f91400cfe00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 --2- 172.21.15.136:0/1465796303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91400b5490 0x7f91400d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 >> 172.21.15.136:0/1465796303 conn(0x7f914006d6c0 msgr2=0x7f91400cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:00.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 shutdown_connections 2024-04-25T11:26:00.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.502+0000 7f9144d88640 1 -- 172.21.15.136:0/1465796303 wait complete. 2024-04-25T11:26:00.518 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-25T11:26:00.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:26:00.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:00.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.670+0000 7f8f70aed640 1 Processor -- start 2024-04-25T11:26:00.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.670+0000 7f8f70aed640 1 -- start start 2024-04-25T11:26:00.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f70aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f6c167a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f70aed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6c15c6f0 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f70aed640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6c15c990 con 0x7f8f6c15ff60 2024-04-25T11:26:00.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f70aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c0c6310 con 0x7f8f6c160370 2024-04-25T11:26:00.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8f6c15c0d0 0x7f8f6c0cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50342/0 (socket says 172.21.15.136:50342) 2024-04-25T11:26:00.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6a59c640 1 -- 172.21.15.136:0/2873297290 learned_addr learned my addr 172.21.15.136:0/2873297290 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:26:00.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6ad9d640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f6c167a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028226451 0 0) 0x7f8f6c15c6f0 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f54003530 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1770942588 0 0) 0x7f8f54003530 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 msgr2=0x7f8f6c167a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f6c167a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 >> v1:172.21.15.157:6789/0 conn(0x7f8f6c15ff60 legacy=0x7f8f6c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c15cc30 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8f5c001680 con 0x7f8f6c15c0d0 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 >> v1:172.21.15.136:6789/0 conn(0x7f8f6c15c0d0 legacy=0x7f8f6c0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f540039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f54004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f54004d50 0x7f8f54005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c0c6310 con 0x7f8f54003f70 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5400bb60 con 0x7f8f54004d50 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5400bd60 con 0x7f8f6c160370 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f54004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6a59c640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f540039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f6ad9d640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f54004d50 0x7f8f54005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 -- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 msgr2=0x7f8f540039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c160370 0x7f8f540039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 -- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f54004d50 msgr2=0x7f8f54005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f54004d50 0x7f8f54005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.674+0000 7f8f69d9b640 1 -- 172.21.15.136:0/2873297290 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f54003530 con 0x7f8f54003f70 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f54004810 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8f60001780 tx=0x7f8f6001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b263cb0421be7f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f600013a0 con 0x7f8f54003f70 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f600052f0 con 0x7f8f54003f70 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f6959a640 1 -- 172.21.15.136:0/2873297290 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60003840 con 0x7f8f54003f70 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- 172.21.15.136:0/2873297290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 msgr2=0x7f8f54004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2873297290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f54004810 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8f60001780 tx=0x7f8f6001c420 comp rx=0 tx=0).stop 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f6ad9d640 1 -- 172.21.15.136:0/2873297290 reap_dead start 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- 172.21.15.136:0/2873297290 shutdown_connections 2024-04-25T11:26:00.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- 172.21.15.136:0/2873297290 >> 172.21.15.136:0/2873297290 conn(0x7f8f6c06f9d0 msgr2=0x7f8f6c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- 172.21.15.136:0/2873297290 shutdown_connections 2024-04-25T11:26:00.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- 172.21.15.136:0/2873297290 wait complete. 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 Processor -- start 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.678+0000 7f8f70aed640 1 -- start start 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c14c240 0x7f8f6c1e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 0x7f8f6c148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c15cf10 con 0x7f8f54003f70 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c15fa40 con 0x7f8f6c0c7f80 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c0c9170 con 0x7f8f6c14c240 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40112/0 (socket says 172.21.15.136:40112) 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 -- 172.21.15.136:0/2707051350 learned_addr learned my addr 172.21.15.136:0/2707051350 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c14c240 0x7f8f6c1e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6ad9d640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 0x7f8f6c148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 -- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c14c240 msgr2=0x7f8f6c1e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c14c240 0x7f8f6c1e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 -- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 msgr2=0x7f8f6c148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 0x7f8f6c148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 -- 172.21.15.136:0/2707051350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c0c77c0 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6ad9d640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 0x7f8f6c148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f6a59c640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c011840 tx=0x7f8f5c011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c010040 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c01a070 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6c154d50 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00fdc0 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6c0c8570 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f6c06a350 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.682+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8f5c01f460 con 0x7f8f54003f70 2024-04-25T11:26:00.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.686+0000 7f8f5affd640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 0x7f8f4c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.686+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f8f5c092830 con 0x7f8f54003f70 2024-04-25T11:26:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.686+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 0x7f8f4c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.686+0000 7f8f69d9b640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 0x7f8f4c071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8f600ac000 tx=0x7f8f600011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:00.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.690+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8f5c05f390 con 0x7f8f54003f70 2024-04-25T11:26:00.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.950+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f6c1e8480 con 0x7f8f54003f70 2024-04-25T11:26:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.950+0000 7f8f5affd640 1 -- 172.21.15.136:0/2707051350 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6093 (secure 0 0 0) 0x7f8f5c05ec90 con 0x7f8f54003f70 2024-04-25T11:26:00.947 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:00.947 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:25:58.820536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:26:00.947 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 26 2024-04-25T11:26:00.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 msgr2=0x7f8f4c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 0x7f8f4c071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8f600ac000 tx=0x7f8f600011a0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 msgr2=0x7f8f6c154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c011840 tx=0x7f8f5c011d00 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 shutdown_connections 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6c0c7f80 0x7f8f6c148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8f4c06f400 0x7f8f4c071910 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f6c14c240 0x7f8f6c1e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 --2- 172.21.15.136:0/2707051350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f54003f70 0x7f8f6c154810 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 >> 172.21.15.136:0/2707051350 conn(0x7f8f6c06f9d0 msgr2=0x7f8f6c15a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 shutdown_connections 2024-04-25T11:26:00.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:00.954+0000 7f8f70aed640 1 -- 172.21.15.136:0/2707051350 wait complete. 2024-04-25T11:26:00.970 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4260": { "addr": "172.21.15.136:6837/1368493675", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1368493675, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1368493675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:25:58.820536+0000", "qdb_cluster": [ 4260, 4269, 4297, 4303 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 4260, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:26:02.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:03.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.142+0000 7f460a48c640 1 Processor -- start 2024-04-25T11:26:03.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.142+0000 7f460a48c640 1 -- start start 2024-04-25T11:26:03.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f460a48c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4604154060 0x7f46040c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f460a48c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f460415a5f0 0x7f46040c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f460a48c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46040c8d40 con 0x7f460415aa00 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f460a48c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f46040c8fd0 con 0x7f460415a5f0 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f460a48c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f46040c3880 con 0x7f4604154060 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f460415a5f0 0x7f46040c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f460415a5f0 0x7f46040c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40126/0 (socket says 172.21.15.136:40126) 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 -- 172.21.15.136:0/1438345668 learned_addr learned my addr 172.21.15.136:0/1438345668 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f4603fff640 1 --2- 172.21.15.136:0/1438345668 >> v2:172.21.15.157:3300/0 conn(0x7f4604154060 0x7f46040c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 -- 172.21.15.136:0/1438345668 >> v2:172.21.15.157:3300/0 conn(0x7f4604154060 msgr2=0x7f46040c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:03.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 --2- 172.21.15.136:0/1438345668 >> v2:172.21.15.157:3300/0 conn(0x7f4604154060 0x7f46040c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 -- 172.21.15.136:0/1438345668 >> v1:172.21.15.157:6789/0 conn(0x7f460415aa00 legacy=0x7f460415d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 -- 172.21.15.136:0/1438345668 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460406d9b0 con 0x7f460415a5f0 2024-04-25T11:26:03.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f46037fe640 1 --2- 172.21.15.136:0/1438345668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f460415a5f0 0x7f46040c71b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f46040c7c60 tx=0x7f45f801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f2c39ec04a6031 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f4602ffd640 1 -- 172.21.15.136:0/1438345668 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f80b5070 con 0x7f460415a5f0 2024-04-25T11:26:03.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f4602ffd640 1 -- 172.21.15.136:0/1438345668 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f80a8a10 con 0x7f460415a5f0 2024-04-25T11:26:03.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.146+0000 7f4602ffd640 1 -- 172.21.15.136:0/1438345668 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f80a8de0 con 0x7f460415a5f0 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- 172.21.15.136:0/1438345668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f460415a5f0 msgr2=0x7f46040c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 --2- 172.21.15.136:0/1438345668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f460415a5f0 0x7f46040c71b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f46040c7c60 tx=0x7f45f801ed10 comp rx=0 tx=0).stop 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- 172.21.15.136:0/1438345668 shutdown_connections 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 --2- 172.21.15.136:0/1438345668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f460415a5f0 0x7f46040c71b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 --2- 172.21.15.136:0/1438345668 >> v2:172.21.15.157:3300/0 conn(0x7f4604154060 0x7f46040c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- 172.21.15.136:0/1438345668 >> 172.21.15.136:0/1438345668 conn(0x7f46040ca5a0 msgr2=0x7f46040ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- 172.21.15.136:0/1438345668 shutdown_connections 2024-04-25T11:26:03.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- 172.21.15.136:0/1438345668 wait complete. 2024-04-25T11:26:03.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 Processor -- start 2024-04-25T11:26:03.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.150+0000 7f460a48c640 1 -- start start 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f460415a5f0 0x7f4604148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 0x7f4604074330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f460406df30 con 0x7f4604154060 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4604154470 con 0x7f460415a5f0 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f460415a0d0 con 0x7f460415aa00 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:40132/0 (socket says 172.21.15.136:40132) 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 -- 172.21.15.136:0/1353246114 learned_addr learned my addr 172.21.15.136:0/1353246114 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f46037fe640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f460415a5f0 0x7f4604148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4608a29640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 0x7f4604074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 -- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 msgr2=0x7f4604074330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 0x7f4604074330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 -- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f460415a5f0 msgr2=0x7f4604148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f460415a5f0 0x7f4604148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 -- 172.21.15.136:0/1353246114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4604151ad0 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4608a29640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 0x7f4604074330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4603fff640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e870 tx=0x7f45f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4017070 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f400c940 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f400ccf0 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4604151db0 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4604152370 con 0x7f4604154060 2024-04-25T11:26:03.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.154+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460406a350 con 0x7f4604154060 2024-04-25T11:26:03.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.158+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f45f401a070 con 0x7f4604154060 2024-04-25T11:26:03.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.158+0000 7f4600ff9640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 0x7f45f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:03.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.158+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f45f408f8c0 con 0x7f4604154060 2024-04-25T11:26:03.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.158+0000 7f46037fe640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 0x7f45f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:03.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.158+0000 7f46037fe640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 0x7f45f0071910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45f80b2000 tx=0x7f45f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:03.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.162+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45f405c420 con 0x7f4604154060 2024-04-25T11:26:03.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46040cc090 con 0x7f4604154060 2024-04-25T11:26:03.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f4600ff9640 1 -- 172.21.15.136:0/1353246114 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6093 (secure 0 0 0) 0x7f45f405bd20 con 0x7f4604154060 2024-04-25T11:26:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:03.380 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:25:58.820536+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:26:03.381 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 26 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 msgr2=0x7f45f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 0x7f45f0071910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45f80b2000 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 msgr2=0x7f460414e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f45f400e870 tx=0x7f45f400ec70 comp rx=0 tx=0).stop 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 shutdown_connections 2024-04-25T11:26:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f45f006f400 0x7f45f0071910 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f460415aa00 0x7f4604074330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f460415a5f0 0x7f4604148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 --2- 172.21.15.136:0/1353246114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4604154060 0x7f460414e550 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 >> 172.21.15.136:0/1353246114 conn(0x7f46040ca5a0 msgr2=0x7f4604075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 shutdown_connections 2024-04-25T11:26:03.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:03.386+0000 7f460a48c640 1 -- 172.21.15.136:0/1353246114 wait complete. 2024-04-25T11:26:03.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:05.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:05.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 Processor -- start 2024-04-25T11:26:05.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 -- start start 2024-04-25T11:26:05.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff878160370 0x7ff8780ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87815c6f0 con 0x7ff87815c0d0 2024-04-25T11:26:05.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff8780c6310 con 0x7ff878160370 2024-04-25T11:26:05.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff87e1ef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff87815c990 con 0x7ff87815ff60 2024-04-25T11:26:05.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff86ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff877fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff878160370 0x7ff8780ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff86ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:55478/0 (socket says 172.21.15.136:55478) 2024-04-25T11:26:05.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.602+0000 7ff86ffff640 1 -- 172.21.15.136:0/3466529934 learned_addr learned my addr 172.21.15.136:0/3466529934 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:05.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff86ffff640 1 -- 172.21.15.136:0/3466529934 >> v2:172.21.15.157:3300/0 conn(0x7ff878160370 msgr2=0x7ff8780ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:05.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3466529934 >> v2:172.21.15.157:3300/0 conn(0x7ff878160370 0x7ff8780ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff86ffff640 1 -- 172.21.15.136:0/3466529934 >> v1:172.21.15.157:6789/0 conn(0x7ff87815c0d0 legacy=0x7ff8780cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff86ffff640 1 -- 172.21.15.136:0/3466529934 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87815cc30 con 0x7ff87815ff60 2024-04-25T11:26:05.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3466529934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff860009870 tx=0x7ff86001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41da023d982baa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff876ffd640 1 -- 172.21.15.136:0/3466529934 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8600ab650 con 0x7ff87815ff60 2024-04-25T11:26:05.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff876ffd640 1 -- 172.21.15.136:0/3466529934 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8600b0070 con 0x7ff87815ff60 2024-04-25T11:26:05.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.606+0000 7ff876ffd640 1 -- 172.21.15.136:0/3466529934 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8600aba10 con 0x7ff87815ff60 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3466529934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 msgr2=0x7ff8780ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3466529934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff860009870 tx=0x7ff86001c560 comp rx=0 tx=0).stop 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3466529934 shutdown_connections 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3466529934 >> v2:172.21.15.157:3300/0 conn(0x7ff878160370 0x7ff8780ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3466529934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff8780ce260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3466529934 >> 172.21.15.136:0/3466529934 conn(0x7ff87806f9d0 msgr2=0x7ff87806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3466529934 shutdown_connections 2024-04-25T11:26:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3466529934 wait complete. 2024-04-25T11:26:05.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 Processor -- start 2024-04-25T11:26:05.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.610+0000 7ff87e1ef640 1 -- start start 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff878145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff878160370 0x7ff8781457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780c9170 con 0x7ff87815c0d0 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff8780cabc0 con 0x7ff878160370 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff878157b40 con 0x7ff87815ff60 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33582/0 (socket says 172.21.15.136:33582) 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 -- 172.21.15.136:0/3185426258 learned_addr learned my addr 172.21.15.136:0/3185426258 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff878145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 -- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 msgr2=0x7ff878145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff878145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 -- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff878160370 msgr2=0x7ff8781457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff878160370 0x7ff8781457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 -- 172.21.15.136:0/3185426258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87815d290 con 0x7ff87815c0d0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff878145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff8777fe640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c3c0 tx=0x7ff86800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868017070 con 0x7ff87815c0d0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8680052d0 con 0x7ff87815c0d0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868003840 con 0x7ff87815c0d0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87815f020 con 0x7ff87815c0d0 2024-04-25T11:26:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.614+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8781fde90 con 0x7ff87815c0d0 2024-04-25T11:26:05.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.618+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87806a350 con 0x7ff87815c0d0 2024-04-25T11:26:05.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.618+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff86801a070 con 0x7ff87815c0d0 2024-04-25T11:26:05.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.618+0000 7ff874ff9640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 0x7ff858071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:05.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.618+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7ff86808ea80 con 0x7ff87815c0d0 2024-04-25T11:26:05.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.618+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 0x7ff858071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:05.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.622+0000 7ff86ffff640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 0x7ff858071910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff8600af000 tx=0x7ff860008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:05.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.626+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff868005480 con 0x7ff87815c0d0 2024-04-25T11:26:05.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.846+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8780ca8b0 con 0x7ff87815c0d0 2024-04-25T11:26:05.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.846+0000 7ff874ff9640 1 -- 172.21.15.136:0/3185426258 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7ff86805b4d0 con 0x7ff87815c0d0 2024-04-25T11:26:05.843 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:05.843 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:03.831757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:resolve","state_seq":34,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:26:05.843 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 27 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 msgr2=0x7ff858071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 0x7ff858071910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff8600af000 tx=0x7ff860008040 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 msgr2=0x7ff8781fd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c3c0 tx=0x7ff86800c880 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 shutdown_connections 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff878160370 0x7ff8781457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff85806f400 0x7ff858071910 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff87815ff60 0x7ff878145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 --2- 172.21.15.136:0/3185426258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff87815c0d0 0x7ff8781fd530 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 >> 172.21.15.136:0/3185426258 conn(0x7ff87806f9d0 msgr2=0x7ff87806fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:05.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 shutdown_connections 2024-04-25T11:26:05.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:05.850+0000 7ff87e1ef640 1 -- 172.21.15.136:0/3185426258 wait complete. 2024-04-25T11:26:05.865 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:07.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:08.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 Processor -- start 2024-04-25T11:26:08.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 -- start start 2024-04-25T11:26:08.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6b140c8cd0 0x7f6b1414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ce260 0x7f6b140ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b140cbad0 con 0x7f6b140ca010 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6b140c8550 con 0x7f6b140c8cd0 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b1a273640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6b140c7c20 con 0x7f6b140ce260 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b13fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6b140c8cd0 0x7f6b1414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b137fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ce260 0x7f6b140ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b13fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6b140c8cd0 0x7f6b1414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:42378/0 (socket says 172.21.15.136:42378) 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b18810640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6b140ca010 0x7f6b140ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:34980/0 (socket says 172.21.15.136:34980) 2024-04-25T11:26:08.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.038+0000 7f6b13fff640 1 -- 172.21.15.136:0/2683391352 learned_addr learned my addr 172.21.15.136:0/2683391352 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b13fff640 1 -- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ce260 msgr2=0x7f6b140ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b13fff640 1 --2- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ce260 0x7f6b140ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b13fff640 1 -- 172.21.15.136:0/2683391352 >> v1:172.21.15.157:6789/0 conn(0x7f6b140ca010 legacy=0x7f6b140ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b13fff640 1 -- 172.21.15.136:0/2683391352 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b140c7f00 con 0x7f6b140c8cd0 2024-04-25T11:26:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b13fff640 1 --2- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1414f680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b140c86d0 tx=0x7f6b0401f200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9faedcc0a3850221 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b12ffd640 1 -- 172.21.15.136:0/2683391352 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b040b4070 con 0x7f6b140c8cd0 2024-04-25T11:26:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b12ffd640 1 -- 172.21.15.136:0/2683391352 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b040af400 con 0x7f6b140c8cd0 2024-04-25T11:26:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b12ffd640 1 -- 172.21.15.136:0/2683391352 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b040af7d0 con 0x7f6b140c8cd0 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 -- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 msgr2=0x7f6b1414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1414f680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b140c86d0 tx=0x7f6b0401f200 comp rx=0 tx=0).stop 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 -- 172.21.15.136:0/2683391352 shutdown_connections 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ce260 0x7f6b140ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2683391352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1414f680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 -- 172.21.15.136:0/2683391352 >> 172.21.15.136:0/2683391352 conn(0x7f6b1406f9d0 msgr2=0x7f6b1406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 -- 172.21.15.136:0/2683391352 shutdown_connections 2024-04-25T11:26:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.042+0000 7f6b1a273640 1 -- 172.21.15.136:0/2683391352 wait complete. 2024-04-25T11:26:08.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.046+0000 7f6b1a273640 1 Processor -- start 2024-04-25T11:26:08.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.046+0000 7f6b1a273640 1 -- start start 2024-04-25T11:26:08.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ca010 0x7f6b1413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b140cb720 con 0x7f6b140ce260 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1414fbc0 con 0x7f6b140c8cd0 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6b141e3d00 con 0x7f6b140ca010 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33588/0 (socket says 172.21.15.136:33588) 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 -- 172.21.15.136:0/2321224814 learned_addr learned my addr 172.21.15.136:0/2321224814 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b137fe640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ca010 0x7f6b1413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b13fff640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 -- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ca010 msgr2=0x7f6b1413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ca010 0x7f6b1413fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 -- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 msgr2=0x7f6b1413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 -- 172.21.15.136:0/2321224814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1414b3a0 con 0x7f6b140ce260 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b18810640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c3c0 tx=0x7f6afc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0074e0 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6afc007680 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0049f0 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b141efe70 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b14154080 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1406a350 con 0x7f6b140ce260 2024-04-25T11:26:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.050+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6afc01a020 con 0x7f6b140ce260 2024-04-25T11:26:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.054+0000 7f6b10ff9640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 0x7f6af4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.054+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f6afc08ee40 con 0x7f6b140ce260 2024-04-25T11:26:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.054+0000 7f6b13fff640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 0x7f6af4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.054+0000 7f6b13fff640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 0x7f6af4071940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b040b1000 tx=0x7f6b040aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:08.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.058+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6afc05b910 con 0x7f6b140ce260 2024-04-25T11:26:08.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.290+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b141403a0 con 0x7f6b140ce260 2024-04-25T11:26:08.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.290+0000 7f6b10ff9640 1 -- 172.21.15.136:0/2321224814 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6095 (secure 0 0 0) 0x7f6afc05b210 con 0x7f6b140ce260 2024-04-25T11:26:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:08.285 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:03.831757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:resolve","state_seq":34,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:26:08.285 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 27 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.290+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 msgr2=0x7f6af4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.290+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 0x7f6af4071940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6b040b1000 tx=0x7f6b040aa000 comp rx=0 tx=0).stop 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 msgr2=0x7f6b14155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c3c0 tx=0x7f6afc00c880 comp rx=0 tx=0).stop 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 shutdown_connections 2024-04-25T11:26:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b140ce260 0x7f6b14155920 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6af406f430 0x7f6af4071940 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6b140ca010 0x7f6b1413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 --2- 172.21.15.136:0/2321224814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6b140c8cd0 0x7f6b1413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 >> 172.21.15.136:0/2321224814 conn(0x7f6b1406f9d0 msgr2=0x7f6b140ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 shutdown_connections 2024-04-25T11:26:08.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:08.294+0000 7f6b1a273640 1 -- 172.21.15.136:0/2321224814 wait complete. 2024-04-25T11:26:08.307 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:10.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.498+0000 7f1e459a5640 1 Processor -- start 2024-04-25T11:26:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.498+0000 7f1e459a5640 1 -- start start 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e459a5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1e4014f840 0x7f1e40074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e459a5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1e400c4340 0x7f1e40073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e459a5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e400c3750 con 0x7f1e40073200 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e459a5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1e4006dc00 con 0x7f1e400c4340 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e459a5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1e4014fc50 con 0x7f1e4014f840 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1e400c4340 0x7f1e40073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1e400c4340 0x7f1e40073610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33598/0 (socket says 172.21.15.136:33598) 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3e7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1e40073200 0x7f1e400c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36734/0 (socket says 172.21.15.136:36734) 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/2241953257 learned_addr learned my addr 172.21.15.136:0/2241953257 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3effd640 1 --2- 172.21.15.136:0/2241953257 >> v2:172.21.15.157:3300/0 conn(0x7f1e4014f840 0x7f1e40074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/2241953257 >> v2:172.21.15.157:3300/0 conn(0x7f1e4014f840 msgr2=0x7f1e40074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 --2- 172.21.15.136:0/2241953257 >> v2:172.21.15.157:3300/0 conn(0x7f1e4014f840 0x7f1e40074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/2241953257 >> v1:172.21.15.136:6789/0 conn(0x7f1e40073200 legacy=0x7f1e400c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/2241953257 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e400c3190 con 0x7f1e400c4340 2024-04-25T11:26:10.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.502+0000 7f1e3dffb640 1 -- 172.21.15.136:0/2241953257 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650970733 0 0) 0x7f1e400c3750 con 0x7f1e40073200 2024-04-25T11:26:10.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e3f7fe640 1 --2- 172.21.15.136:0/2241953257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e400c4340 0x7f1e40073610 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d150 tx=0x7f1e3801ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932750ae27a8bec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e3dffb640 1 -- 172.21.15.136:0/2241953257 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800b2d0 con 0x7f1e400c4340 2024-04-25T11:26:10.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e3dffb640 1 -- 172.21.15.136:0/2241953257 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3800b470 con 0x7f1e400c4340 2024-04-25T11:26:10.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e3dffb640 1 -- 172.21.15.136:0/2241953257 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e380a79e0 con 0x7f1e400c4340 2024-04-25T11:26:10.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- 172.21.15.136:0/2241953257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e400c4340 msgr2=0x7f1e40073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 --2- 172.21.15.136:0/2241953257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e400c4340 0x7f1e40073610 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d150 tx=0x7f1e3801ebf0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- 172.21.15.136:0/2241953257 shutdown_connections 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 --2- 172.21.15.136:0/2241953257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e400c4340 0x7f1e40073610 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 --2- 172.21.15.136:0/2241953257 >> v2:172.21.15.157:3300/0 conn(0x7f1e4014f840 0x7f1e40074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- 172.21.15.136:0/2241953257 >> 172.21.15.136:0/2241953257 conn(0x7f1e400ca5a0 msgr2=0x7f1e400ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- 172.21.15.136:0/2241953257 shutdown_connections 2024-04-25T11:26:10.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- 172.21.15.136:0/2241953257 wait complete. 2024-04-25T11:26:10.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 Processor -- start 2024-04-25T11:26:10.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.506+0000 7f1e459a5640 1 -- start start 2024-04-25T11:26:10.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40073200 0x7f1e4014ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400c4340 0x7f1e401558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1e401597f0 con 0x7f1e4014f840 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e40153380 con 0x7f1e40073200 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1e40159970 con 0x7f1e400c4340 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33600/0 (socket says 172.21.15.136:33600) 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/3271078507 learned_addr learned my addr 172.21.15.136:0/3271078507 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3effd640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40073200 0x7f1e4014ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400c4340 msgr2=0x7f1e401558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400c4340 0x7f1e401558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40073200 msgr2=0x7f1e4014ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40073200 0x7f1e4014ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 -- 172.21.15.136:0/3271078507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4014a370 con 0x7f1e4014f840 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e3f7fe640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1e3801b800 tx=0x7f1e3801b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800b720 con 0x7f1e4014f840 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e380c0070 con 0x7f1e4014f840 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4014a0d0 con 0x7f1e4014f840 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38009a00 con 0x7f1e4014f840 2024-04-25T11:26:10.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e401f4840 con 0x7f1e4014f840 2024-04-25T11:26:10.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.510+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4006a350 con 0x7f1e4014f840 2024-04-25T11:26:10.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.514+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e3800b2e0 con 0x7f1e4014f840 2024-04-25T11:26:10.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.514+0000 7f1e27fff640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 0x7f1e2c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:10.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.514+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f1e3812c7c0 con 0x7f1e4014f840 2024-04-25T11:26:10.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.514+0000 7f1e3effd640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 0x7f1e2c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:10.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.514+0000 7f1e3effd640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 0x7f1e2c071910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b930 tx=0x7f1e30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:10.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.518+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1e380187e0 con 0x7f1e4014f840 2024-04-25T11:26:10.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.726+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4014e660 con 0x7f1e4014f840 2024-04-25T11:26:10.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.726+0000 7f1e27fff640 1 -- 172.21.15.136:0/3271078507 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6097 (secure 0 0 0) 0x7f1e380f9290 con 0x7f1e4014f840 2024-04-25T11:26:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:10.723 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:08.998590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:26:10.723 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 28 2024-04-25T11:26:10.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 msgr2=0x7f1e2c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 0x7f1e2c071910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b930 tx=0x7f1e30008040 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 msgr2=0x7f1e40155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1e3801b800 tx=0x7f1e3801b990 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 shutdown_connections 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e4014f840 0x7f1e40155e30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1e2c06f400 0x7f1e2c071910 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400c4340 0x7f1e401558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 --2- 172.21.15.136:0/3271078507 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40073200 0x7f1e4014ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 >> 172.21.15.136:0/3271078507 conn(0x7f1e400ca5a0 msgr2=0x7f1e400c8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 shutdown_connections 2024-04-25T11:26:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:10.730+0000 7f1e459a5640 1 -- 172.21.15.136:0/3271078507 wait complete. 2024-04-25T11:26:10.748 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:12.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:12.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.930+0000 7fe88d909640 1 Processor -- start 2024-04-25T11:26:12.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.930+0000 7fe88d909640 1 -- start start 2024-04-25T11:26:12.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe88d909640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe8880c9b50 0x7fe88815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe88d909640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 0x7fe88815c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe88d909640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8880c8a20 con 0x7fe8880ca010 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe88d909640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe88806c670 con 0x7fe8880c9b50 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe88d909640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe8880c8d00 con 0x7fe88806d210 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe8880c9b50 0x7fe88815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe8880c9b50 0x7fe88815bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33616/0 (socket says 172.21.15.136:33616) 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe886ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe8880ca010 0x7fe88806cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36738/0 (socket says 172.21.15.136:36738) 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 -- 172.21.15.136:0/1916230732 learned_addr learned my addr 172.21.15.136:0/1916230732 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8877fe640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 0x7fe88815c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 -- 172.21.15.136:0/1916230732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 msgr2=0x7fe88815c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 0x7fe88815c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 -- 172.21.15.136:0/1916230732 >> v1:172.21.15.136:6789/0 conn(0x7fe8880ca010 legacy=0x7fe88806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 -- 172.21.15.136:0/1916230732 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe888074d10 con 0x7fe8880c9b50 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe885ffb640 1 -- 172.21.15.136:0/1916230732 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190267319 0 0) 0x7fe8880c8a20 con 0x7fe8880ca010 2024-04-25T11:26:12.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8877fe640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 0x7fe88815c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:12.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe8867fc640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe88815bfe0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009870 tx=0x7fe87c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792fe84a8bb90996 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:12.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe885ffb640 1 -- 172.21.15.136:0/1916230732 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0b2070 con 0x7fe8880c9b50 2024-04-25T11:26:12.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe885ffb640 1 -- 172.21.15.136:0/1916230732 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87c0a9070 con 0x7fe8880c9b50 2024-04-25T11:26:12.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.934+0000 7fe885ffb640 1 -- 172.21.15.136:0/1916230732 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0ad880 con 0x7fe8880c9b50 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- 172.21.15.136:0/1916230732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 msgr2=0x7fe88815bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe88815bfe0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009870 tx=0x7fe87c01c560 comp rx=0 tx=0).stop 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- 172.21.15.136:0/1916230732 shutdown_connections 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe88806d210 0x7fe88815c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- 172.21.15.136:0/1916230732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe88815bfe0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- 172.21.15.136:0/1916230732 >> 172.21.15.136:0/1916230732 conn(0x7fe8880ca5a0 msgr2=0x7fe8880ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- 172.21.15.136:0/1916230732 shutdown_connections 2024-04-25T11:26:12.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- 172.21.15.136:0/1916230732 wait complete. 2024-04-25T11:26:12.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 Processor -- start 2024-04-25T11:26:12.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- start start 2024-04-25T11:26:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 0x7fe8881f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 0x7fe8880cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881694d0 con 0x7fe8880c9b50 2024-04-25T11:26:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.938+0000 7fe88d909640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe888169350 con 0x7fe88806d210 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe88d909640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe888169650 con 0x7fe8880ca010 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33622/0 (socket says 172.21.15.136:33622) 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 -- 172.21.15.136:0/2832111182 learned_addr learned my addr 172.21.15.136:0/2832111182 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe886ffd640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 0x7fe8881f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8877fe640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 0x7fe8880cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 -- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 msgr2=0x7fe8880cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 0x7fe8880cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 -- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 msgr2=0x7fe8881f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 0x7fe8881f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 -- 172.21.15.136:0/2832111182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881f6d40 con 0x7fe8880c9b50 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe886ffd640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 0x7fe8881f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:12.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8877fe640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 0x7fe8880cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe8867fc640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0b0040 tx=0x7fe87c019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0bc070 con 0x7fe8880c9b50 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87c0b6640 con 0x7fe8880c9b50 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe888153fd0 con 0x7fe8880c9b50 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0b2040 con 0x7fe8880c9b50 2024-04-25T11:26:12.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe888155320 con 0x7fe8880c9b50 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe888155560 con 0x7fe8880c9b50 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe87c0bf070 con 0x7fe8880c9b50 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 0x7fe874071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7fe87c128cf0 con 0x7fe8880c9b50 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.942+0000 7fe886ffd640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 0x7fe874071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.946+0000 7fe886ffd640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 0x7fe874071940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe878000f70 tx=0x7fe878009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:12.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:12.950+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe888155560 con 0x7fe8880c9b50 2024-04-25T11:26:13.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe88806a350 con 0x7fe8880c9b50 2024-04-25T11:26:13.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe86ffff640 1 -- 172.21.15.136:0/2832111182 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6097 (secure 0 0 0) 0x7fe87c12d070 con 0x7fe8880c9b50 2024-04-25T11:26:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:13.173 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:08.998590+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303]},"id":1}]} 2024-04-25T11:26:13.173 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 28 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 msgr2=0x7fe874071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 0x7fe874071940 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe878000f70 tx=0x7fe878009040 comp rx=0 tx=0).stop 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 msgr2=0x7fe888153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0b0040 tx=0x7fe87c019300 comp rx=0 tx=0).stop 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 shutdown_connections 2024-04-25T11:26:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe87406f430 0x7fe874071940 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.178+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8880ca010 0x7fe8880cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.182+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8880c9b50 0x7fe888153a90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.182+0000 7fe88d909640 1 --2- 172.21.15.136:0/2832111182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88806d210 0x7fe8881f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.182+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 >> 172.21.15.136:0/2832111182 conn(0x7fe8880ca5a0 msgr2=0x7fe88815b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.182+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 shutdown_connections 2024-04-25T11:26:13.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:13.182+0000 7fe88d909640 1 -- 172.21.15.136:0/2832111182 wait complete. 2024-04-25T11:26:13.196 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4260": { "addr": "172.21.15.136:6837/1368493675", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1368493675, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1368493675, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4260, "incarnation": 16, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 15 }, "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:26:08.998590+0000", "qdb_cluster": [ 4260, 4269, 4297, 4303 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 4260, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:26:15.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:15.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 Processor -- start 2024-04-25T11:26:15.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 -- start start 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415ff60 0x7f77940ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7794160370 0x7f77940ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f779415c6f0 con 0x7f779415c0d0 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f77940c6310 con 0x7f7794160370 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779bb90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f779415c990 con 0x7f779415ff60 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7794160370 0x7f77940ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7794160370 0x7f77940ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36752/0 (socket says 172.21.15.136:36752) 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 -- 172.21.15.136:0/694185854 learned_addr learned my addr 172.21.15.136:0/694185854 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779912b640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415ff60 0x7f77940ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 -- 172.21.15.136:0/694185854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415ff60 msgr2=0x7f77940ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:15.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415ff60 0x7f77940ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 -- 172.21.15.136:0/694185854 >> v1:172.21.15.136:6789/0 conn(0x7f779415c0d0 legacy=0x7f77940cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.370+0000 7f779a12d640 1 -- 172.21.15.136:0/694185854 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779415cc30 con 0x7f7794160370 2024-04-25T11:26:15.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779a12d640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7794160370 0x7f77940ce7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f778c009e50 tx=0x7f778c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6d5231a8a9b744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:15.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779892a640 1 -- 172.21.15.136:0/694185854 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c0a57f0 con 0x7f7794160370 2024-04-25T11:26:15.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779892a640 1 -- 172.21.15.136:0/694185854 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f778c0a5990 con 0x7f7794160370 2024-04-25T11:26:15.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779892a640 1 -- 172.21.15.136:0/694185854 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c0a5d60 con 0x7f7794160370 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 -- 172.21.15.136:0/694185854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7794160370 msgr2=0x7f77940ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7794160370 0x7f77940ce7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f778c009e50 tx=0x7f778c01ca30 comp rx=0 tx=0).stop 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 -- 172.21.15.136:0/694185854 shutdown_connections 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7794160370 0x7f77940ce7a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 --2- 172.21.15.136:0/694185854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415ff60 0x7f77940ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 -- 172.21.15.136:0/694185854 >> 172.21.15.136:0/694185854 conn(0x7f779406f9d0 msgr2=0x7f779406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 -- 172.21.15.136:0/694185854 shutdown_connections 2024-04-25T11:26:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.374+0000 7f779bb90640 1 -- 172.21.15.136:0/694185854 wait complete. 2024-04-25T11:26:15.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 Processor -- start 2024-04-25T11:26:15.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- start start 2024-04-25T11:26:15.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 0x7f77940c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 0x7f7794143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f77940c9170 con 0x7f779415ff60 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f77940cabc0 con 0x7f7794160370 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77940c75d0 con 0x7f779415c0d0 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36764/0 (socket says 172.21.15.136:36764) 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 -- 172.21.15.136:0/111744555 learned_addr learned my addr 172.21.15.136:0/111744555 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779992c640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 0x7f77940c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779a12d640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 0x7f7794143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 -- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 msgr2=0x7f77940c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 0x7f77940c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 -- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 msgr2=0x7f7794143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 0x7f7794143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 -- 172.21.15.136:0/111744555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77940c7100 con 0x7f779415ff60 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779a12d640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 0x7f7794143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779992c640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 0x7f77940c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779912b640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f779000d620 tx=0x7f779000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779000b620 con 0x7f779415ff60 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7790019070 con 0x7f779415ff60 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7794144e40 con 0x7f779415ff60 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779000b9e0 con 0x7f779415ff60 2024-04-25T11:26:15.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.378+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77941453c0 con 0x7f779415ff60 2024-04-25T11:26:15.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f779406a350 con 0x7f779415ff60 2024-04-25T11:26:15.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f779000bb80 con 0x7f779415ff60 2024-04-25T11:26:15.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f77867fc640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 0x7f7778071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:15.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f779008fbe0 con 0x7f779415ff60 2024-04-25T11:26:15.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f779992c640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 0x7f7778071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:15.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.382+0000 7f779992c640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 0x7f7778071910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7788001650 tx=0x7f7788009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:15.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.386+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f779005c630 con 0x7f779415ff60 2024-04-25T11:26:15.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.614+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f779406e070 con 0x7f779415ff60 2024-04-25T11:26:15.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.614+0000 7f77867fc640 1 -- 172.21.15.136:0/111744555 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6094 (secure 0 0 0) 0x7f779005bf30 con 0x7f779415ff60 2024-04-25T11:26:15.609 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:15.609 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:14.023066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:26:15.609 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 29 2024-04-25T11:26:15.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 msgr2=0x7f7778071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 0x7f7778071910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7788001650 tx=0x7f7788009040 comp rx=0 tx=0).stop 2024-04-25T11:26:15.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 msgr2=0x7f77940cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f779000d620 tx=0x7f779000dae0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 shutdown_connections 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7794160370 0x7f7794143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f779415ff60 0x7f77940cd840 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f777806f400 0x7f7778071910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 --2- 172.21.15.136:0/111744555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f779415c0d0 0x7f77940c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 >> 172.21.15.136:0/111744555 conn(0x7f779406f9d0 msgr2=0x7f7794159db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 shutdown_connections 2024-04-25T11:26:15.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:15.618+0000 7f779bb90640 1 -- 172.21.15.136:0/111744555 wait complete. 2024-04-25T11:26:15.632 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:17.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:17.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.798+0000 7ffa087a6640 1 Processor -- start 2024-04-25T11:26:17.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.798+0000 7ffa087a6640 1 -- start start 2024-04-25T11:26:17.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa087a6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffa00160370 0x7ffa00167a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa087a6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0015c6f0 con 0x7ffa0015c0d0 2024-04-25T11:26:17.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa087a6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0015c990 con 0x7ffa0015ff60 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa087a6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffa000c6310 con 0x7ffa00160370 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06542640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffa0015c0d0 0x7ffa000cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:34324/0 (socket says 172.21.15.136:34324) 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06542640 1 -- 172.21.15.136:0/130755082 learned_addr learned my addr 172.21.15.136:0/130755082 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06d43640 1 --2- 172.21.15.136:0/130755082 >> v2:172.21.15.157:3300/0 conn(0x7ffa00160370 0x7ffa00167a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699885963 0 0) 0x7ffa0015c6f0 con 0x7ffa0015c0d0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ec003530 con 0x7ffa0015c0d0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 861660026 0 0) 0x7ff9ec003530 con 0x7ffa0015c0d0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 >> v2:172.21.15.157:3300/0 conn(0x7ffa00160370 msgr2=0x7ffa00167a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 --2- 172.21.15.136:0/130755082 >> v2:172.21.15.157:3300/0 conn(0x7ffa00160370 0x7ffa00167a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 >> v1:172.21.15.157:6789/0 conn(0x7ffa0015ff60 legacy=0x7ffa000ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0015cc30 con 0x7ffa0015c0d0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7ff9fc001680 con 0x7ffa0015c0d0 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 >> v1:172.21.15.136:6789/0 conn(0x7ffa0015c0d0 legacy=0x7ffa000cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9ec0039c0 0x7ff9ec003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ff9ec004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9ec0051d0 0x7ff9ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa000c6310 con 0x7ff9ec004810 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec00c170 con 0x7ff9ec0039c0 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec00c370 con 0x7ff9ec0051d0 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ff9ec004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06542640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9ec0039c0 0x7ff9ec003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06d43640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9ec0051d0 0x7ff9ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 -- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9ec0051d0 msgr2=0x7ff9ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9ec0051d0 0x7ff9ec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 -- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9ec0039c0 msgr2=0x7ff9ec003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9ec0039c0 0x7ff9ec003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa05d41640 1 -- 172.21.15.136:0/130755082 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec003530 con 0x7ff9ec004810 2024-04-25T11:26:17.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.802+0000 7ffa06d43640 1 -- 172.21.15.136:0/130755082 reap_dead start 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa05d41640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ff9ec004c20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0001750 tx=0x7ff9f001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a971e9984927b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00013a0 con 0x7ff9ec004810 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f00052d0 con 0x7ff9ec004810 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa05540640 1 -- 172.21.15.136:0/130755082 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00054c0 con 0x7ff9ec004810 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 -- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 msgr2=0x7ff9ec004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ff9ec004c20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0001750 tx=0x7ff9f001c3c0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 -- 172.21.15.136:0/130755082 shutdown_connections 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 --2- 172.21.15.136:0/130755082 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ff9ec004c20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 -- 172.21.15.136:0/130755082 >> 172.21.15.136:0/130755082 conn(0x7ffa0006f9d0 msgr2=0x7ffa0006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 -- 172.21.15.136:0/130755082 shutdown_connections 2024-04-25T11:26:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.806+0000 7ffa087a6640 1 -- 172.21.15.136:0/130755082 wait complete. 2024-04-25T11:26:17.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 Processor -- start 2024-04-25T11:26:17.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- start start 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 0x7ffa000cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa001e8760 0x7ffa001e8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0015cf10 con 0x7ff9ec004810 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0015fa40 con 0x7ffa001e8760 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa000c9170 con 0x7ffa000c7f70 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36770/0 (socket says 172.21.15.136:36770) 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 -- 172.21.15.136:0/1296519732 learned_addr learned my addr 172.21.15.136:0/1296519732 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa05d41640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 0x7ffa000cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06d43640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa001e8760 0x7ffa001e8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 -- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 msgr2=0x7ffa000cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 0x7ffa000cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 -- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa001e8760 msgr2=0x7ffa001e8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa001e8760 0x7ffa001e8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 -- 172.21.15.136:0/1296519732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa000c86c0 con 0x7ff9ec004810 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa05d41640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 0x7ffa000cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa06542640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0117d0 tx=0x7ff9fc011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00f8e0 con 0x7ff9ec004810 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9fc01b070 con 0x7ff9ec004810 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa000c7960 con 0x7ff9ec004810 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc00fcc0 con 0x7ff9ec004810 2024-04-25T11:26:17.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.810+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa001437a0 con 0x7ff9ec004810 2024-04-25T11:26:17.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0006a350 con 0x7ff9ec004810 2024-04-25T11:26:17.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff9fc020460 con 0x7ff9ec004810 2024-04-25T11:26:17.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ff9f6ffd640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 0x7ff9e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:17.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7ff9fc0928f0 con 0x7ff9ec004810 2024-04-25T11:26:17.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ffa05d41640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 0x7ff9e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:17.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.814+0000 7ffa05d41640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 0x7ff9e4071910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa0015cb60 tx=0x7ff9f00011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:17.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:17.818+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff9fc05f3c0 con 0x7ff9ec004810 2024-04-25T11:26:18.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.062+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa000caf90 con 0x7ff9ec004810 2024-04-25T11:26:18.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.062+0000 7ff9f6ffd640 1 -- 172.21.15.136:0/1296519732 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6094 (secure 0 0 0) 0x7ff9fc05ecc0 con 0x7ff9ec004810 2024-04-25T11:26:18.059 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:18.059 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:14.023066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:26:18.059 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 29 2024-04-25T11:26:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 msgr2=0x7ff9e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 0x7ff9e4071910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa0015cb60 tx=0x7ff9f00011a0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 msgr2=0x7ffa00151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0117d0 tx=0x7ff9fc011c90 comp rx=0 tx=0).stop 2024-04-25T11:26:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 shutdown_connections 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa001e8760 0x7ffa001e8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9e406f400 0x7ff9e4071910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa000c7f70 0x7ffa000cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 --2- 172.21.15.136:0/1296519732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9ec004810 0x7ffa00151650 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 >> 172.21.15.136:0/1296519732 conn(0x7ffa0006f9d0 msgr2=0x7ffa000c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 shutdown_connections 2024-04-25T11:26:18.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:18.066+0000 7ffa087a6640 1 -- 172.21.15.136:0/1296519732 wait complete. 2024-04-25T11:26:18.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:26:20.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:20.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.250+0000 7f1b39143640 1 Processor -- start 2024-04-25T11:26:20.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.250+0000 7f1b39143640 1 -- start start 2024-04-25T11:26:20.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b39143640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415ff60 0x7f1b340ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b39143640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b34160370 0x7f1b340ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b39143640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3415c6f0 con 0x7f1b3415c0d0 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b39143640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1b340c6310 con 0x7f1b34160370 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b39143640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3415c990 con 0x7f1b3415ff60 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b34160370 0x7f1b340ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b34160370 0x7f1b340ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36786/0 (socket says 172.21.15.136:36786) 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 -- 172.21.15.136:0/2923465433 learned_addr learned my addr 172.21.15.136:0/2923465433 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3259c640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415ff60 0x7f1b340ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 -- 172.21.15.136:0/2923465433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415ff60 msgr2=0x7f1b340ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415ff60 0x7f1b340ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 -- 172.21.15.136:0/2923465433 >> v1:172.21.15.157:6789/0 conn(0x7f1b3415c0d0 legacy=0x7f1b340cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 -- 172.21.15.136:0/2923465433 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3415cc30 con 0x7f1b34160370 2024-04-25T11:26:20.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b3359e640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b34160370 0x7f1b340ce7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009e50 tx=0x7f1b2c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b801a8380c2dd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:20.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b31d9b640 1 -- 172.21.15.136:0/2923465433 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0a57f0 con 0x7f1b34160370 2024-04-25T11:26:20.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b31d9b640 1 -- 172.21.15.136:0/2923465433 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c0a5990 con 0x7f1b34160370 2024-04-25T11:26:20.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.254+0000 7f1b31d9b640 1 -- 172.21.15.136:0/2923465433 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0a5d60 con 0x7f1b34160370 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- 172.21.15.136:0/2923465433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b34160370 msgr2=0x7f1b340ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b34160370 0x7f1b340ce7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009e50 tx=0x7f1b2c01ca30 comp rx=0 tx=0).stop 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- 172.21.15.136:0/2923465433 shutdown_connections 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b34160370 0x7f1b340ce7a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- 172.21.15.136:0/2923465433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415ff60 0x7f1b340ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- 172.21.15.136:0/2923465433 >> 172.21.15.136:0/2923465433 conn(0x7f1b3406f9d0 msgr2=0x7f1b3406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- 172.21.15.136:0/2923465433 shutdown_connections 2024-04-25T11:26:20.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- 172.21.15.136:0/2923465433 wait complete. 2024-04-25T11:26:20.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 Processor -- start 2024-04-25T11:26:20.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- start start 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415c0d0 0x7f1b3414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b34160370 0x7f1b3415eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1b340c9170 con 0x7f1b3415ff60 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1b340cabc0 con 0x7f1b34160370 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b39143640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b340c7330 con 0x7f1b3415c0d0 2024-04-25T11:26:20.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b3259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b3259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36800/0 (socket says 172.21.15.136:36800) 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.258+0000 7f1b3259c640 1 -- 172.21.15.136:0/1460311281 learned_addr learned my addr 172.21.15.136:0/1460311281 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b32d9d640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415c0d0 0x7f1b3414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3359e640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b34160370 0x7f1b3415eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 -- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415c0d0 msgr2=0x7f1b3414fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415c0d0 0x7f1b3414fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 -- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b34160370 msgr2=0x7f1b3415eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b34160370 0x7f1b3415eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 -- 172.21.15.136:0/1460311281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b340c6e60 con 0x7f1b3415ff60 2024-04-25T11:26:20.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b3259c640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800e8a0 tx=0x7f1b2800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28017070 con 0x7f1b3415ff60 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2800c940 con 0x7f1b3415ff60 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b341f55f0 con 0x7f1b3415ff60 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800ccf0 con 0x7f1b3415ff60 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b341f5bb0 con 0x7f1b3415ff60 2024-04-25T11:26:20.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3406a350 con 0x7f1b3415ff60 2024-04-25T11:26:20.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1b2801a070 con 0x7f1b3415ff60 2024-04-25T11:26:20.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 0x7f1b20071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:20.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f1b2808f5b0 con 0x7f1b3415ff60 2024-04-25T11:26:20.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.262+0000 7f1b32d9d640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 0x7f1b20071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:20.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.266+0000 7f1b32d9d640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 0x7f1b20071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1b24001770 tx=0x7f1b24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:20.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.270+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1b2805b9a0 con 0x7f1b3415ff60 2024-04-25T11:26:20.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.494+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b34155ab0 con 0x7f1b3415ff60 2024-04-25T11:26:20.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b1b7fe640 1 -- 172.21.15.136:0/1460311281 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6099 (secure 0 0 0) 0x7f1b2805fca0 con 0x7f1b3415ff60 2024-04-25T11:26:20.491 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:20.491 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:26:20.491 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 30 2024-04-25T11:26:20.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 msgr2=0x7f1b20071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 0x7f1b20071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1b24001770 tx=0x7f1b24009040 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 msgr2=0x7f1b340c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800e8a0 tx=0x7f1b2800eca0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 shutdown_connections 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b34160370 0x7f1b3415eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b3415ff60 0x7f1b340c6920 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b2006f400 0x7f1b20071910 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 --2- 172.21.15.136:0/1460311281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b3415c0d0 0x7f1b3414fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 >> 172.21.15.136:0/1460311281 conn(0x7f1b3406f9d0 msgr2=0x7f1b340ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 shutdown_connections 2024-04-25T11:26:20.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:20.498+0000 7f1b39143640 1 -- 172.21.15.136:0/1460311281 wait complete. 2024-04-25T11:26:20.519 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=4, replacing gid=4254 2024-04-25T11:26:20.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2024-04-25T11:26:20.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:26:20.813 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:26:27.520 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-25T11:26:27.521 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T11:26:27.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T11:26:27.524 INFO:tasks.ceph.mds.a:Started 2024-04-25T11:26:27.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:27.639 INFO:tasks.ceph.mds.a.smithi136.stdout:starting mds.a at 2024-04-25T11:26:27.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 Processor -- start 2024-04-25T11:26:27.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 -- start start 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540ce260 0x7f14540ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f14540ca010 0x7f14540ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14540cbad0 con 0x7f14540c8cd0 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f14540c8550 con 0x7f14540ca010 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f145b532640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14540c7c20 con 0x7f14540ce260 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1459acf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f14540ca010 0x7f14540ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f14592ce640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f14540c8cd0 0x7f145414f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60242/0 (socket says 172.21.15.136:60242) 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f14592ce640 1 -- 172.21.15.136:0/4058923625 learned_addr learned my addr 172.21.15.136:0/4058923625 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1458acd640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540ce260 0x7f14540ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1459acf640 1 -- 172.21.15.136:0/4058923625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540ce260 msgr2=0x7f14540ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1459acf640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540ce260 0x7f14540ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1459acf640 1 -- 172.21.15.136:0/4058923625 >> v1:172.21.15.136:6789/0 conn(0x7f14540c8cd0 legacy=0x7f145414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f1459acf640 1 -- 172.21.15.136:0/4058923625 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14540c7f00 con 0x7f14540ca010 2024-04-25T11:26:27.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.702+0000 7f144bfff640 1 -- 172.21.15.136:0/4058923625 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580253327 0 0) 0x7f14540cbad0 con 0x7f14540c8cd0 2024-04-25T11:26:27.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f1459acf640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14540ca420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f14540c86d0 tx=0x7f144c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccc5dd36718c604 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:27.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f144bfff640 1 -- 172.21.15.136:0/4058923625 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0b4070 con 0x7f14540ca010 2024-04-25T11:26:27.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f144bfff640 1 -- 172.21.15.136:0/4058923625 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c0af600 con 0x7f14540ca010 2024-04-25T11:26:27.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f144bfff640 1 -- 172.21.15.136:0/4058923625 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c0af9d0 con 0x7f14540ca010 2024-04-25T11:26:27.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 -- 172.21.15.136:0/4058923625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 msgr2=0x7f14540ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14540ca420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f14540c86d0 tx=0x7f144c01f130 comp rx=0 tx=0).stop 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 -- 172.21.15.136:0/4058923625 shutdown_connections 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14540ca420 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 --2- 172.21.15.136:0/4058923625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540ce260 0x7f14540ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 -- 172.21.15.136:0/4058923625 >> 172.21.15.136:0/4058923625 conn(0x7f145406f9d0 msgr2=0x7f145406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 -- 172.21.15.136:0/4058923625 shutdown_connections 2024-04-25T11:26:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.706+0000 7f145b532640 1 -- 172.21.15.136:0/4058923625 wait complete. 2024-04-25T11:26:27.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 Processor -- start 2024-04-25T11:26:27.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- start start 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540c8cd0 0x7f14541540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14540ce260 0x7f1454155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f14540cb720 con 0x7f14540ca010 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14540ca960 con 0x7f14540ce260 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f145414b070 con 0x7f14540c8cd0 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51822/0 (socket says 172.21.15.136:51822) 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 -- 172.21.15.136:0/2885084311 learned_addr learned my addr 172.21.15.136:0/2885084311 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:27.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f14592ce640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540c8cd0 0x7f14541540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1459acf640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14540ce260 0x7f1454155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 -- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540c8cd0 msgr2=0x7f14541540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540c8cd0 0x7f14541540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 -- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14540ce260 msgr2=0x7f1454155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14540ce260 0x7f1454155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 -- 172.21.15.136:0/2885084311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145414bb60 con 0x7f14540ca010 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1458acd640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f144400d620 tx=0x7f144400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144400b620 con 0x7f14540ca010 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1444019070 con 0x7f14540ca010 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144400b9e0 con 0x7f14540ca010 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14541f40e0 con 0x7f14540ca010 2024-04-25T11:26:27.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.710+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145414b5a0 con 0x7f14540ca010 2024-04-25T11:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145406a350 con 0x7f14540ca010 2024-04-25T11:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f144400bb80 con 0x7f14540ca010 2024-04-25T11:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f1449ffb640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 0x7f1438071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f144408fc10 con 0x7f14540ca010 2024-04-25T11:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f14592ce640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 0x7f1438071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:27.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.714+0000 7f14592ce640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 0x7f1438071910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1450000f70 tx=0x7f1450007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:27.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.718+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f144405c650 con 0x7f14540ca010 2024-04-25T11:26:27.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.962+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f145406fde0 con 0x7f14540ca010 2024-04-25T11:26:27.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f1449ffb640 1 -- 172.21.15.136:0/2885084311 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6099 (secure 0 0 0) 0x7f144405bf50 con 0x7f14540ca010 2024-04-25T11:26:27.959 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:27.959 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4260,4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:26:27.960 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 30 2024-04-25T11:26:27.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 msgr2=0x7f1438071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 0x7f1438071910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1450000f70 tx=0x7f1450007c10 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 msgr2=0x7f14541f3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f144400d620 tx=0x7f144400dae0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 shutdown_connections 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14540ce260 0x7f1454155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14540ca010 0x7f14541f3ba0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f143806f400 0x7f1438071910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 --2- 172.21.15.136:0/2885084311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14540c8cd0 0x7f14541540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 >> 172.21.15.136:0/2885084311 conn(0x7f145406f9d0 msgr2=0x7f1454075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 shutdown_connections 2024-04-25T11:26:27.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:27.966+0000 7f145b532640 1 -- 172.21.15.136:0/2885084311 wait complete. 2024-04-25T11:26:27.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T11:26:29.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:26:30.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.130+0000 7f0212f6d640 1 Processor -- start 2024-04-25T11:26:30.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.130+0000 7f0212f6d640 1 -- start start 2024-04-25T11:26:30.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0212f6d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f020c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0212f6d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020c0c9af0 con 0x7f020c0c96e0 2024-04-25T11:26:30.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0212f6d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020c0cb220 con 0x7f020c0c79e0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0212f6d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f020c06cc10 con 0x7f020c15aae0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0210d09640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f020c0c96e0 0x7f020c1540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60258/0 (socket says 172.21.15.136:60258) 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0210d09640 1 -- 172.21.15.136:0/781728484 learned_addr learned my addr 172.21.15.136:0/781728484 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f020c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2674994071 0 0) 0x7f020c0c9af0 con 0x7f020c0c96e0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01fc003530 con 0x7f020c0c96e0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215153040 0 0) 0x7f01fc003530 con 0x7f020c0c96e0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 msgr2=0x7f020c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f020c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 >> v1:172.21.15.157:6789/0 conn(0x7f020c0c79e0 legacy=0x7f020c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c0cb500 con 0x7f020c0c96e0 2024-04-25T11:26:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f02000016a0 con 0x7f020c0c96e0 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 >> v1:172.21.15.136:6789/0 conn(0x7f020c0c96e0 legacy=0x7f020c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f01fc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01fc003f70 0x7f01fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01fc004d50 0x7f01fc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f020c06cc10 con 0x7f01fc004d50 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc00bb60 con 0x7f01fc003f70 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc00bd60 con 0x7f020c15aae0 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01fc004d50 0x7f01fc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f0210d09640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f01fc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 -- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 msgr2=0x7f01fc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020bfff640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01fc003f70 0x7f01fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c15aae0 0x7f01fc0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 -- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01fc003f70 msgr2=0x7f01fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01fc003f70 0x7f01fc004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 -- 172.21.15.136:0/781728484 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc003530 con 0x7f01fc004d50 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f021150a640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01fc004d50 0x7f01fc005160 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0204009f90 tx=0x7f020401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb2c20d04fc9419 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02040b2070 con 0x7f01fc004d50 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02040a5c70 con 0x7f01fc004d50 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.134+0000 7f020b7fe640 1 -- 172.21.15.136:0/781728484 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204009c10 con 0x7f01fc004d50 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- 172.21.15.136:0/781728484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01fc004d50 msgr2=0x7f01fc005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 --2- 172.21.15.136:0/781728484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01fc004d50 0x7f01fc005160 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0204009f90 tx=0x7f020401cc60 comp rx=0 tx=0).stop 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f021150a640 1 -- 172.21.15.136:0/781728484 reap_dead start 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- 172.21.15.136:0/781728484 shutdown_connections 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- 172.21.15.136:0/781728484 >> 172.21.15.136:0/781728484 conn(0x7f020c06f9d0 msgr2=0x7f020c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- 172.21.15.136:0/781728484 shutdown_connections 2024-04-25T11:26:30.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- 172.21.15.136:0/781728484 wait complete. 2024-04-25T11:26:30.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 Processor -- start 2024-04-25T11:26:30.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- start start 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c146e50 0x7f020c1f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c140f70 0x7f020c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f020c0ced40 con 0x7f020c1f4550 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f020c0cf0f0 con 0x7f020c140f70 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0212f6d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f020c1f3b80 con 0x7f020c146e50 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f020bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f020bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51840/0 (socket says 172.21.15.136:51840) 2024-04-25T11:26:30.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f020bfff640 1 -- 172.21.15.136:0/2938491663 learned_addr learned my addr 172.21.15.136:0/2938491663 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f0210d09640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c146e50 0x7f020c1f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.138+0000 7f021150a640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c140f70 0x7f020c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 -- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c146e50 msgr2=0x7f020c1f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c146e50 0x7f020c1f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 -- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c140f70 msgr2=0x7f020c155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c140f70 0x7f020c155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 -- 172.21.15.136:0/2938491663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c154a50 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f020bfff640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f01f800d570 tx=0x7f01f800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8003620 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f8017070 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80039c0 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020c154d30 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020c1f4960 con 0x7f020c1f4550 2024-04-25T11:26:30.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f020c06a350 con 0x7f020c1f4550 2024-04-25T11:26:30.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f01f8003b60 con 0x7f020c1f4550 2024-04-25T11:26:30.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 0x7f01f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:26:30.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f01f808f470 con 0x7f020c1f4550 2024-04-25T11:26:30.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.142+0000 7f0210d09640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 0x7f01f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:26:30.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.146+0000 7f0210d09640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 0x7f01f0071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f020c0c8530 tx=0x7f0200007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:26:30.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.150+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f01f805c000 con 0x7f020c1f4550 2024-04-25T11:26:30.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.382+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f020c06e450 con 0x7f020c1f4550 2024-04-25T11:26:30.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.382+0000 7f02097fa640 1 -- 172.21.15.136:0/2938491663 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7f01f805b900 con 0x7f020c1f4550 2024-04-25T11:26:30.378 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:26:30.378 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:26:30.378 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:26:30.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 msgr2=0x7f01f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 0x7f01f0071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f020c0c8530 tx=0x7f0200007040 comp rx=0 tx=0).stop 2024-04-25T11:26:30.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 msgr2=0x7f020c140a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f01f800d570 tx=0x7f01f800da30 comp rx=0 tx=0).stop 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 shutdown_connections 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f020c140f70 0x7f020c155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f020c1f4550 0x7f020c140a30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f01f006f400 0x7f01f0071910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 --2- 172.21.15.136:0/2938491663 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f020c146e50 0x7f020c1f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 >> 172.21.15.136:0/2938491663 conn(0x7f020c06f9d0 msgr2=0x7f020c152d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 shutdown_connections 2024-04-25T11:26:30.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:26:30.386+0000 7f0212f6d640 1 -- 172.21.15.136:0/2938491663 wait complete. 2024-04-25T11:26:30.398 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-25T11:26:30.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 105 secs before thrashing 2024-04-25T11:26:51.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:26:51.234 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:27:21.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:27:21.291 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:27:51.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:27:51.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:28:15.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:15.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.580+0000 7fce4f646640 1 Processor -- start 2024-04-25T11:28:15.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.580+0000 7fce4f646640 1 -- start start 2024-04-25T11:28:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4f646640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fce4806d210 0x7fce4815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4f646640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce480c8a20 con 0x7fce480c9b50 2024-04-25T11:28:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4f646640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce480c8d00 con 0x7fce480ca010 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4f646640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fce4806c670 con 0x7fce4806d210 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4cbe1640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fce480c9b50 0x7fce4815bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47578/0 (socket says 172.21.15.136:47578) 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4cbe1640 1 -- 172.21.15.136:0/3663833764 learned_addr learned my addr 172.21.15.136:0/3663833764 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3663833764 >> v2:172.21.15.157:3300/0 conn(0x7fce4806d210 0x7fce4815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006637618 0 0) 0x7fce480c8a20 con 0x7fce480c9b50 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce34003530 con 0x7fce480c9b50 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4060674767 0 0) 0x7fce34003530 con 0x7fce480c9b50 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 >> v2:172.21.15.157:3300/0 conn(0x7fce4806d210 msgr2=0x7fce4815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 --2- 172.21.15.136:0/3663833764 >> v2:172.21.15.157:3300/0 conn(0x7fce4806d210 0x7fce4815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 >> v1:172.21.15.157:6789/0 conn(0x7fce480ca010 legacy=0x7fce4806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce48074d10 con 0x7fce480c9b50 2024-04-25T11:28:15.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fce380016a0 con 0x7fce480c9b50 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 >> v1:172.21.15.136:6789/0 conn(0x7fce480c9b50 legacy=0x7fce4815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce340039c0 0x7fce34003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce34004810 0x7fce34004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce4806c670 con 0x7fce340051d0 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce3400c170 con 0x7fce34004810 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce3400c370 con 0x7fce340039c0 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4d3e2640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce340039c0 0x7fce34003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4cbe1640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce34004810 0x7fce34004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 -- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce340039c0 msgr2=0x7fce34003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce340039c0 0x7fce34003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 -- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce34004810 msgr2=0x7fce34004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:15.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce34004810 0x7fce34004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4dbe3640 1 -- 172.21.15.136:0/3663833764 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce34003530 con 0x7fce340051d0 2024-04-25T11:28:15.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.584+0000 7fce4d3e2640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce340039c0 0x7fce34003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:15.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4dbe3640 1 -- 172.21.15.136:0/3663833764 reap_dead start 2024-04-25T11:28:15.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce3400b9d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fce40009fc0 tx=0x7fce4001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf9c2f4c4f8d9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:15.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40009c10 con 0x7fce340051d0 2024-04-25T11:28:15.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce400a5c50 con 0x7fce340051d0 2024-04-25T11:28:15.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce3ffff640 1 -- 172.21.15.136:0/3663833764 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400039a0 con 0x7fce340051d0 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 -- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 msgr2=0x7fce3400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce3400b9d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fce40009fc0 tx=0x7fce4001cc00 comp rx=0 tx=0).stop 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 -- 172.21.15.136:0/3663833764 shutdown_connections 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 --2- 172.21.15.136:0/3663833764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce3400b9d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 -- 172.21.15.136:0/3663833764 >> 172.21.15.136:0/3663833764 conn(0x7fce480ca5a0 msgr2=0x7fce480ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 -- 172.21.15.136:0/3663833764 shutdown_connections 2024-04-25T11:28:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.588+0000 7fce4f646640 1 -- 172.21.15.136:0/3663833764 wait complete. 2024-04-25T11:28:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 Processor -- start 2024-04-25T11:28:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 -- start start 2024-04-25T11:28:15.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce4815ff40 0x7fce481536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce4815f100 0x7fce4815f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce4806cb00 con 0x7fce340051d0 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce480c9240 con 0x7fce4815f100 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4f646640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce4806cc80 con 0x7fce4815ff40 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41528/0 (socket says 172.21.15.136:41528) 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 -- 172.21.15.136:0/3112660219 learned_addr learned my addr 172.21.15.136:0/3112660219 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4cbe1640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce4815ff40 0x7fce481536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4dbe3640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce4815f100 0x7fce4815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 -- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce4815ff40 msgr2=0x7fce481536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce4815ff40 0x7fce481536e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 -- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce4815f100 msgr2=0x7fce4815f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce4815f100 0x7fce4815f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.592+0000 7fce4d3e2640 1 -- 172.21.15.136:0/3112660219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce48150840 con 0x7fce340051d0 2024-04-25T11:28:15.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce4d3e2640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e4d0 tx=0x7fce4400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce440052e0 con 0x7fce340051d0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce44018070 con 0x7fce340051d0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce481f2070 con 0x7fce340051d0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44005480 con 0x7fce340051d0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce481f2630 con 0x7fce340051d0 2024-04-25T11:28:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.596+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce481f2cf0 con 0x7fce340051d0 2024-04-25T11:28:15.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.600+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fce44003820 con 0x7fce340051d0 2024-04-25T11:28:15.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.600+0000 7fce3dffb640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 0x7fce2c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:15.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.600+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7fce4408f8d0 con 0x7fce340051d0 2024-04-25T11:28:15.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.600+0000 7fce4cbe1640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 0x7fce2c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:15.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.600+0000 7fce4cbe1640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 0x7fce2c071940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce481f6420 tx=0x7fce38007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.604+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce481f2cf0 con 0x7fce340051d0 2024-04-25T11:28:15.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.836+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce4806a350 con 0x7fce340051d0 2024-04-25T11:28:15.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.836+0000 7fce3dffb640 1 -- 172.21.15.136:0/3112660219 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7fce4405be30 con 0x7fce340051d0 2024-04-25T11:28:15.827 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:15.828 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:15.828 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 msgr2=0x7fce2c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 0x7fce2c071940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce481f6420 tx=0x7fce38007040 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 msgr2=0x7fce48150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fce4400e4d0 tx=0x7fce4400e990 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 shutdown_connections 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce4815f100 0x7fce4815f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce2c06f430 0x7fce2c071940 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce4815ff40 0x7fce481536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 --2- 172.21.15.136:0/3112660219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce340051d0 0x7fce48150300 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:15.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 >> 172.21.15.136:0/3112660219 conn(0x7fce480ca5a0 msgr2=0x7fce480cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:15.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 shutdown_connections 2024-04-25T11:28:15.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:15.840+0000 7fce4f646640 1 -- 172.21.15.136:0/3112660219 wait complete. 2024-04-25T11:28:15.850 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=3) 2024-04-25T11:28:15.850 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-25T11:28:15.850 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:28:15.865 INFO:tasks.ceph.mds.e:Stopped 2024-04-25T11:28:15.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.056+0000 7ff32e7c5640 1 Processor -- start 2024-04-25T11:28:16.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.056+0000 7ff32e7c5640 1 -- start start 2024-04-25T11:28:16.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff328156750 0x7ff3280ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff3280ca7d0 0x7ff3280cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 0x7ff32806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff32815c410 con 0x7ff3280ca7d0 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff3280cfde0 con 0x7ff3280cf9d0 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32e7c5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff32815c6b0 con 0x7ff328156750 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff3280ca7d0 0x7ff3280cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff3280ca7d0 0x7ff3280cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41536/0 (socket says 172.21.15.136:41536) 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 -- 172.21.15.136:0/3702784210 learned_addr learned my addr 172.21.15.136:0/3702784210 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32cd62640 1 --2- 172.21.15.136:0/3702784210 >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 0x7ff32806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff327fff640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff328156750 0x7ff3280ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:16.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 -- 172.21.15.136:0/3702784210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff328156750 msgr2=0x7ff3280ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:16.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff328156750 0x7ff3280ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 -- 172.21.15.136:0/3702784210 >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 msgr2=0x7ff32806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:16.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 --2- 172.21.15.136:0/3702784210 >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 0x7ff32806e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 -- 172.21.15.136:0/3702784210 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32815c990 con 0x7ff3280ca7d0 2024-04-25T11:28:16.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff32cd62640 1 --2- 172.21.15.136:0/3702784210 >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 0x7ff32806e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:16.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.060+0000 7ff31ffff640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3280ca7d0 0x7ff3280cf490 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff328160810 tx=0x7ff31001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc4f3cec3b488bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:16.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff3277fe640 1 -- 172.21.15.136:0/3702784210 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3100b0070 con 0x7ff3280ca7d0 2024-04-25T11:28:16.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff3277fe640 1 -- 172.21.15.136:0/3702784210 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3100ab600 con 0x7ff3280ca7d0 2024-04-25T11:28:16.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff3277fe640 1 -- 172.21.15.136:0/3702784210 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3100ab9d0 con 0x7ff3280ca7d0 2024-04-25T11:28:16.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- 172.21.15.136:0/3702784210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3280ca7d0 msgr2=0x7ff3280cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:16.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3280ca7d0 0x7ff3280cf490 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff328160810 tx=0x7ff31001c520 comp rx=0 tx=0).stop 2024-04-25T11:28:16.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- 172.21.15.136:0/3702784210 shutdown_connections 2024-04-25T11:28:16.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3280ca7d0 0x7ff3280cf490 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/3702784210 >> v2:172.21.15.157:3300/0 conn(0x7ff3280cf9d0 0x7ff32806e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/3702784210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff328156750 0x7ff3280ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- 172.21.15.136:0/3702784210 >> 172.21.15.136:0/3702784210 conn(0x7ff328070f70 msgr2=0x7ff328068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- 172.21.15.136:0/3702784210 shutdown_connections 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- 172.21.15.136:0/3702784210 wait complete. 2024-04-25T11:28:16.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 Processor -- start 2024-04-25T11:28:16.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.064+0000 7ff32e7c5640 1 -- start start 2024-04-25T11:28:16.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3280ca7d0 0x7ff328157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3280cf9d0 0x7ff3280b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff32806d9e0 con 0x7ff328158f10 2024-04-25T11:28:16.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff32806d860 con 0x7ff3280cf9d0 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff32815ff80 con 0x7ff3280ca7d0 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41538/0 (socket says 172.21.15.136:41538) 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 -- 172.21.15.136:0/1083450448 learned_addr learned my addr 172.21.15.136:0/1083450448 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 -- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3280ca7d0 msgr2=0x7ff328157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3280ca7d0 0x7ff328157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 -- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3280cf9d0 msgr2=0x7ff3280b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3280cf9d0 0x7ff3280b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 -- 172.21.15.136:0/1083450448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3281580c0 con 0x7ff328158f10 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32cd62640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff320007ec0 tx=0x7ff32000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32001a070 con 0x7ff328158f10 2024-04-25T11:28:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff320015600 con 0x7ff328158f10 2024-04-25T11:28:16.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3200159f0 con 0x7ff328158f10 2024-04-25T11:28:16.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32815b390 con 0x7ff328158f10 2024-04-25T11:28:16.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3281570a0 con 0x7ff328158f10 2024-04-25T11:28:16.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.068+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff32806a350 con 0x7ff328158f10 2024-04-25T11:28:16.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.072+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff32001d070 con 0x7ff328158f10 2024-04-25T11:28:16.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.072+0000 7ff3257fa640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 0x7ff308071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:16.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.072+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7ff32008fce0 con 0x7ff328158f10 2024-04-25T11:28:16.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.072+0000 7ff327fff640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 0x7ff308071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:16.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.072+0000 7ff327fff640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 0x7ff308071940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff31800b7f0 tx=0x7ff318008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:16.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.076+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff320024020 con 0x7ff328158f10 2024-04-25T11:28:16.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.312+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3280cadd0 con 0x7ff328158f10 2024-04-25T11:28:16.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.312+0000 7ff3257fa640 1 -- 172.21.15.136:0/1083450448 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7ff320010070 con 0x7ff328158f10 2024-04-25T11:28:16.303 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:16.303 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:16.303 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:16.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 msgr2=0x7ff308071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 0x7ff308071940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff31800b7f0 tx=0x7ff318008040 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 msgr2=0x7ff328159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff320007ec0 tx=0x7ff32000d4a0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 shutdown_connections 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff328158f10 0x7ff328159320 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3280cf9d0 0x7ff3280b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff30806f430 0x7ff308071940 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 --2- 172.21.15.136:0/1083450448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3280ca7d0 0x7ff328157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 >> 172.21.15.136:0/1083450448 conn(0x7ff328070f70 msgr2=0x7ff328075ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 shutdown_connections 2024-04-25T11:28:16.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:16.316+0000 7ff32e7c5640 1 -- 172.21.15.136:0/1083450448 wait complete. 2024-04-25T11:28:16.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:18.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:18.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 Processor -- start 2024-04-25T11:28:18.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 -- start start 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fedb4156330 0x7fedb415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb415d010 con 0x7fedb40b5ba0 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedb40ccb80 con 0x7fedb40cdee0 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedbc28a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fedb41602f0 con 0x7fedb4156330 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedba827640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fedb40b5ba0 0x7fedb41669d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47592/0 (socket says 172.21.15.136:47592) 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedba827640 1 -- 172.21.15.136:0/2232523460 learned_addr learned my addr 172.21.15.136:0/2232523460 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedba026640 1 --2- 172.21.15.136:0/2232523460 >> v2:172.21.15.157:3300/0 conn(0x7fedb4156330 0x7fedb415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3715355297 0 0) 0x7fedac0014a0 con 0x7fedb40b5ba0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda0003530 con 0x7fedb40b5ba0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3661703403 0 0) 0x7feda0003530 con 0x7fedb40b5ba0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 >> v2:172.21.15.157:3300/0 conn(0x7fedb4156330 msgr2=0x7fedb415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 --2- 172.21.15.136:0/2232523460 >> v2:172.21.15.157:3300/0 conn(0x7fedb4156330 0x7fedb415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 >> v1:172.21.15.157:6789/0 conn(0x7fedb40cdee0 legacy=0x7fedb40b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb4156740 con 0x7fedb40b5ba0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fedac001bd0 con 0x7fedb40b5ba0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 >> v1:172.21.15.136:6789/0 conn(0x7fedb40b5ba0 legacy=0x7fedb41669d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda00039c0 0x7feda0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.484+0000 7fedb9024640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda0004810 0x7feda0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 0x7feda000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41602f0 con 0x7feda00051d0 2024-04-25T11:28:18.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feda000c160 con 0x7feda0004810 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feda000c360 con 0x7feda00039c0 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 0x7feda000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba026640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda00039c0 0x7feda0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9825640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda0004810 0x7feda0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 -- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda00039c0 msgr2=0x7feda0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda00039c0 0x7feda0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 -- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda0004810 msgr2=0x7feda0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda0004810 0x7feda0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 -- 172.21.15.136:0/2232523460 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda0003530 con 0x7feda00051d0 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 -- 172.21.15.136:0/2232523460 reap_dead start 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedba827640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 0x7feda000b9d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedac00aa50 tx=0x7fedac01d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec85bcf0ce62a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac00a670 con 0x7feda00051d0 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedac0ad650 con 0x7feda00051d0 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedb9024640 1 -- 172.21.15.136:0/2232523460 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac0ada40 con 0x7feda00051d0 2024-04-25T11:28:18.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 -- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 msgr2=0x7feda000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 0x7feda000b9d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedac00aa50 tx=0x7fedac01d2e0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 -- 172.21.15.136:0/2232523460 shutdown_connections 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 --2- 172.21.15.136:0/2232523460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda00051d0 0x7feda000b9d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 -- 172.21.15.136:0/2232523460 >> 172.21.15.136:0/2232523460 conn(0x7fedb406d6c0 msgr2=0x7fedb406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 -- 172.21.15.136:0/2232523460 shutdown_connections 2024-04-25T11:28:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.488+0000 7fedbc28a640 1 -- 172.21.15.136:0/2232523460 wait complete. 2024-04-25T11:28:18.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.492+0000 7fedbc28a640 1 Processor -- start 2024-04-25T11:28:18.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.492+0000 7fedbc28a640 1 -- start start 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 0x7fedb415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb40cb4e0 0x7fedb40d1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fedb415fca0 con 0x7fedb40d0370 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fedb40cc9c0 con 0x7feda00051d0 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fedb415ff80 con 0x7fedb40cb4e0 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41570/0 (socket says 172.21.15.136:41570) 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 -- 172.21.15.136:0/73990281 learned_addr learned my addr 172.21.15.136:0/73990281 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedba026640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 0x7fedb415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedba827640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb40cb4e0 0x7fedb40d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 -- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb40cb4e0 msgr2=0x7fedb40d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb40cb4e0 0x7fedb40d1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 -- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 msgr2=0x7fedb415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 0x7fedb415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 -- 172.21.15.136:0/73990281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb41f3000 con 0x7fedb40d0370 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedba026640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 0x7fedb415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedb9825640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feda800e410 tx=0x7feda800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8019070 con 0x7fedb40d0370 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feda80052e0 con 0x7fedb40d0370 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb41f32e0 con 0x7fedb40d0370 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8003820 con 0x7fedb40d0370 2024-04-25T11:28:18.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb40cfe50 con 0x7fedb40d0370 2024-04-25T11:28:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb40cf4f0 con 0x7fedb40d0370 2024-04-25T11:28:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feda801c070 con 0x7fedb40d0370 2024-04-25T11:28:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 0x7fed98071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7feda808ec40 con 0x7fedb40d0370 2024-04-25T11:28:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.496+0000 7fedba026640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 0x7fed98071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:18.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.500+0000 7fedba026640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 0x7fed98071910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009a70 tx=0x7fedb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:18.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.504+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feda805b0c0 con 0x7fedb40d0370 2024-04-25T11:28:18.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.776+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedb406a350 con 0x7fedb40d0370 2024-04-25T11:28:18.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.776+0000 7feda6ffd640 1 -- 172.21.15.136:0/73990281 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7feda805f3c0 con 0x7fedb40d0370 2024-04-25T11:28:18.768 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:18.768 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:18.768 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 msgr2=0x7fed98071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 0x7fed98071910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009a70 tx=0x7fedb0008040 comp rx=0 tx=0).stop 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 msgr2=0x7fedb40cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feda800e410 tx=0x7feda800e8d0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 shutdown_connections 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fed9806f400 0x7fed98071910 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fedb40cb4e0 0x7fedb40d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fedb40d0370 0x7fedb40cafa0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 --2- 172.21.15.136:0/73990281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda00051d0 0x7fedb415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:18.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 >> 172.21.15.136:0/73990281 conn(0x7fedb406d6c0 msgr2=0x7fedb415dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:18.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 shutdown_connections 2024-04-25T11:28:18.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:18.780+0000 7fedbc28a640 1 -- 172.21.15.136:0/73990281 wait complete. 2024-04-25T11:28:18.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:20.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:20.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.972+0000 7fb925ac1640 1 Processor -- start 2024-04-25T11:28:20.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.972+0000 7fb925ac1640 1 -- start start 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb925ac1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9200ce260 0x7fb9200ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb925ac1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200ca010 0x7fb9200ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb925ac1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9200cbad0 con 0x7fb9200c8cd0 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb925ac1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb9200c8550 con 0x7fb9200ce260 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb925ac1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9200c7c20 con 0x7fb9200ca010 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9200ce260 0x7fb9200ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9200ce260 0x7fb9200ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41572/0 (socket says 172.21.15.136:41572) 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 -- 172.21.15.136:0/1614735837 learned_addr learned my addr 172.21.15.136:0/1614735837 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91ffff640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200ca010 0x7fb9200ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 -- 172.21.15.136:0/1614735837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200ca010 msgr2=0x7fb9200ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200ca010 0x7fb9200ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 -- 172.21.15.136:0/1614735837 >> v1:172.21.15.157:6789/0 conn(0x7fb9200c8cd0 legacy=0x7fb92014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:20.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.976+0000 7fb91effd640 1 -- 172.21.15.136:0/1614735837 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9200c7f00 con 0x7fb9200ce260 2024-04-25T11:28:20.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb91effd640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ce670 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c86d0 tx=0x7fb914021400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fbceebc9aefba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:20.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb91e7fc640 1 -- 172.21.15.136:0/1614735837 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9140b6070 con 0x7fb9200ce260 2024-04-25T11:28:20.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb91e7fc640 1 -- 172.21.15.136:0/1614735837 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9140b1600 con 0x7fb9200ce260 2024-04-25T11:28:20.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb91e7fc640 1 -- 172.21.15.136:0/1614735837 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9140b19d0 con 0x7fb9200ce260 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 -- 172.21.15.136:0/1614735837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 msgr2=0x7fb9200ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ce670 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c86d0 tx=0x7fb914021400 comp rx=0 tx=0).stop 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 -- 172.21.15.136:0/1614735837 shutdown_connections 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200ca010 0x7fb9200ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 --2- 172.21.15.136:0/1614735837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ce670 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 -- 172.21.15.136:0/1614735837 >> 172.21.15.136:0/1614735837 conn(0x7fb92006f9d0 msgr2=0x7fb92006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 -- 172.21.15.136:0/1614735837 shutdown_connections 2024-04-25T11:28:20.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.980+0000 7fb925ac1640 1 -- 172.21.15.136:0/1614735837 wait complete. 2024-04-25T11:28:20.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 Processor -- start 2024-04-25T11:28:20.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- start start 2024-04-25T11:28:20.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200c8cd0 0x7fb9201f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9200ca010 0x7fb920151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb9200cb720 con 0x7fb9200ce260 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb9200ca960 con 0x7fb9200ca010 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9201f1fa0 con 0x7fb9200c8cd0 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41580/0 (socket says 172.21.15.136:41580) 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 -- 172.21.15.136:0/80333552 learned_addr learned my addr 172.21.15.136:0/80333552 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91f7fe640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200c8cd0 0x7fb9201f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91effd640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9200ca010 0x7fb920151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 -- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200c8cd0 msgr2=0x7fb9201f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200c8cd0 0x7fb9201f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 -- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9200ca010 msgr2=0x7fb920151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9200ca010 0x7fb920151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:20.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 -- 172.21.15.136:0/80333552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92015b430 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb91ffff640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb918007e90 tx=0x7fb91800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb918014420 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb918019070 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92015b710 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb918014800 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb920158a70 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.984+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92006a350 con 0x7fb9200ce260 2024-04-25T11:28:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.988+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb9180149a0 con 0x7fb9200ce260 2024-04-25T11:28:20.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.988+0000 7fb907fff640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 0x7fb90c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:20.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.988+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7fb91808ff90 con 0x7fb9200ce260 2024-04-25T11:28:20.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.988+0000 7fb91f7fe640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 0x7fb90c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:20.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.988+0000 7fb91f7fe640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 0x7fb90c071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb910000f70 tx=0x7fb910009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:20.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:20.992+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb91805c410 con 0x7fb9200ce260 2024-04-25T11:28:21.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.232+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb92006fe80 con 0x7fb9200ce260 2024-04-25T11:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.232+0000 7fb907fff640 1 -- 172.21.15.136:0/80333552 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7fb918060710 con 0x7fb9200ce260 2024-04-25T11:28:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:21.223 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 msgr2=0x7fb90c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 0x7fb90c071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb910000f70 tx=0x7fb910009040 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 msgr2=0x7fb9200ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb918007e90 tx=0x7fb91800e4b0 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 shutdown_connections 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9200ce260 0x7fb9200ccb00 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9200ca010 0x7fb920151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb90c06f430 0x7fb90c071940 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 --2- 172.21.15.136:0/80333552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9200c8cd0 0x7fb9201f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 >> 172.21.15.136:0/80333552 conn(0x7fb92006f9d0 msgr2=0x7fb92015c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 shutdown_connections 2024-04-25T11:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:21.236+0000 7fb925ac1640 1 -- 172.21.15.136:0/80333552 wait complete. 2024-04-25T11:28:21.253 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:21.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:28:21.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:28:23.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:23.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 Processor -- start 2024-04-25T11:28:23.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 -- start start 2024-04-25T11:28:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f77f814f840 0x7f77f8074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77f8073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f80c3750 con 0x7f77f8073200 2024-04-25T11:28:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f77f806dc00 con 0x7f77f814f840 2024-04-25T11:28:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fec6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77f814fc50 con 0x7f77f80c4340 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77f7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f77f8073200 0x7f77f80c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47604/0 (socket says 172.21.15.136:47604) 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77f7fff640 1 -- 172.21.15.136:0/820960897 learned_addr learned my addr 172.21.15.136:0/820960897 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fd20c640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77f8073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77fca0b640 1 --2- 172.21.15.136:0/820960897 >> v2:172.21.15.157:3300/0 conn(0x7f77f814f840 0x7f77f8074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183323456 0 0) 0x7f77f80c3750 con 0x7f77f8073200 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.436+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77e8003530 con 0x7f77f8073200 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1133222698 0 0) 0x7f77e8003530 con 0x7f77f8073200 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 msgr2=0x7f77f8073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77f8073610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 >> v2:172.21.15.157:3300/0 conn(0x7f77f814f840 msgr2=0x7f77f8074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 --2- 172.21.15.136:0/820960897 >> v2:172.21.15.157:3300/0 conn(0x7f77f814f840 0x7f77f8074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f80c3190 con 0x7f77f8073200 2024-04-25T11:28:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f77e4001680 con 0x7f77f8073200 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 >> v1:172.21.15.136:6789/0 conn(0x7f77f8073200 legacy=0x7f77f80c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e80039c0 0x7f77e8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e8004d50 0x7f77e8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f77f806dc00 con 0x7f77e80039c0 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f77f814fc50 con 0x7f77e8004d50 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77e800b9d0 con 0x7f77f80c4340 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e80039c0 0x7f77e8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f7fff640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 -- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 msgr2=0x7f77e8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fd20c640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e8004d50 0x7f77e8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f80c4340 0x7f77e8004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 -- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e8004d50 msgr2=0x7f77e8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e8004d50 0x7f77e8005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 -- 172.21.15.136:0/820960897 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e8003530 con 0x7f77e80039c0 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77fca0b640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e80039c0 0x7f77e8003e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009ae0 tx=0x7f77ec01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c51c229c6326e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0096b0 con 0x7f77e80039c0 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77ec0a6450 con 0x7f77e80039c0 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.440+0000 7f77f77fe640 1 -- 172.21.15.136:0/820960897 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0a6840 con 0x7f77e80039c0 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- 172.21.15.136:0/820960897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e80039c0 msgr2=0x7f77e8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 --2- 172.21.15.136:0/820960897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e80039c0 0x7f77e8003e30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009ae0 tx=0x7f77ec01ca70 comp rx=0 tx=0).stop 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fd20c640 1 -- 172.21.15.136:0/820960897 reap_dead start 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- 172.21.15.136:0/820960897 shutdown_connections 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- 172.21.15.136:0/820960897 >> 172.21.15.136:0/820960897 conn(0x7f77f80ca5a0 msgr2=0x7f77f80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:23.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- 172.21.15.136:0/820960897 shutdown_connections 2024-04-25T11:28:23.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- 172.21.15.136:0/820960897 wait complete. 2024-04-25T11:28:23.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 Processor -- start 2024-04-25T11:28:23.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- start start 2024-04-25T11:28:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 0x7f77f81f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f81406e0 0x7f77f813fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f77f8153380 con 0x7f77f814e6a0 2024-04-25T11:28:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f77f8159c10 con 0x7f77e80039c0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fec6f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77f81597f0 con 0x7f77f81406e0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fd20c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fd20c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41602/0 (socket says 172.21.15.136:41602) 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77fd20c640 1 -- 172.21.15.136:0/1761779537 learned_addr learned my addr 172.21.15.136:0/1761779537 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.444+0000 7f77f7fff640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f81406e0 0x7f77f813fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fca0b640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 0x7f77f81f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 -- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f81406e0 msgr2=0x7f77f813fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f81406e0 0x7f77f813fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 -- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 msgr2=0x7f77f81f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 0x7f77f81f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 -- 172.21.15.136:0/1761779537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f81f39d0 con 0x7f77f814e6a0 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fca0b640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 0x7f77f81f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fd20c640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f77f80c3a60 tx=0x7f77f000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f001c070 con 0x7f77f814e6a0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f0017400 con 0x7f77f814e6a0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f814abd0 con 0x7f77f814e6a0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f00177f0 con 0x7f77f814e6a0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f8156f60 con 0x7f77f814e6a0 2024-04-25T11:28:23.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f814ecc0 con 0x7f77f814e6a0 2024-04-25T11:28:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f77f001f070 con 0x7f77f814e6a0 2024-04-25T11:28:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 0x7f77dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f77f008f910 con 0x7f77f814e6a0 2024-04-25T11:28:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.448+0000 7f77fca0b640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 0x7f77dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.452+0000 7f77fca0b640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 0x7f77dc071910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0ad000 tx=0x7f77ec0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:23.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.456+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f77f0012070 con 0x7f77f814e6a0 2024-04-25T11:28:23.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.664+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77f806a350 con 0x7f77f814e6a0 2024-04-25T11:28:23.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.664+0000 7f77f57fa640 1 -- 172.21.15.136:0/1761779537 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7f77f005c3e0 con 0x7f77f814e6a0 2024-04-25T11:28:23.656 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:23.656 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:23.656 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:23.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 msgr2=0x7f77dc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 0x7f77dc071910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0ad000 tx=0x7f77ec0046d0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 msgr2=0x7f77f81455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f77f80c3a60 tx=0x7f77f000f4a0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 shutdown_connections 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77f814e6a0 0x7f77f81455c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f77dc06f400 0x7f77dc071910 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77f81406e0 0x7f77f813fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 --2- 172.21.15.136:0/1761779537 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e80039c0 0x7f77f81f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 >> 172.21.15.136:0/1761779537 conn(0x7f77f80ca5a0 msgr2=0x7f77f81494e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 shutdown_connections 2024-04-25T11:28:23.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:23.668+0000 7f77fec6f640 1 -- 172.21.15.136:0/1761779537 wait complete. 2024-04-25T11:28:23.680 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:25.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:25.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.848+0000 7f191e091640 1 Processor -- start 2024-04-25T11:28:25.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.848+0000 7f191e091640 1 -- start start 2024-04-25T11:28:25.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f191e091640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1918074980 0x7f19180c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f191e091640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f191e091640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19180c3750 con 0x7f191814f840 2024-04-25T11:28:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f191e091640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f191806dc00 con 0x7f1918074980 2024-04-25T11:28:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f191e091640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f191814fc50 con 0x7f19180c4170 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:34356/0 (socket says 172.21.15.136:34356) 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1916ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1918074980 0x7f19180c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 -- 172.21.15.136:0/3412500455 learned_addr learned my addr 172.21.15.136:0/3412500455 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 -- 172.21.15.136:0/3412500455 >> v2:172.21.15.157:3300/0 conn(0x7f1918074980 msgr2=0x7f19180c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 --2- 172.21.15.136:0/3412500455 >> v2:172.21.15.157:3300/0 conn(0x7f1918074980 0x7f19180c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 -- 172.21.15.136:0/3412500455 >> v1:172.21.15.157:6789/0 conn(0x7f191814f840 legacy=0x7f1918073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:25.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.852+0000 7f1917fff640 1 -- 172.21.15.136:0/3412500455 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19180c3190 con 0x7f19180c4170 2024-04-25T11:28:25.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f1917fff640 1 --2- 172.21.15.136:0/3412500455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1910009e50 tx=0x7f191001ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf000eb43dbe69b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:25.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f19167fc640 1 -- 172.21.15.136:0/3412500455 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100a57f0 con 0x7f19180c4170 2024-04-25T11:28:25.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f19167fc640 1 -- 172.21.15.136:0/3412500455 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19100a5990 con 0x7f19180c4170 2024-04-25T11:28:25.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f19167fc640 1 -- 172.21.15.136:0/3412500455 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19100a5d60 con 0x7f19180c4170 2024-04-25T11:28:25.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- 172.21.15.136:0/3412500455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 msgr2=0x7f19181636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:25.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 --2- 172.21.15.136:0/3412500455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1910009e50 tx=0x7f191001ca30 comp rx=0 tx=0).stop 2024-04-25T11:28:25.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- 172.21.15.136:0/3412500455 shutdown_connections 2024-04-25T11:28:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 --2- 172.21.15.136:0/3412500455 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f19181636f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 --2- 172.21.15.136:0/3412500455 >> v2:172.21.15.157:3300/0 conn(0x7f1918074980 0x7f19180c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- 172.21.15.136:0/3412500455 >> 172.21.15.136:0/3412500455 conn(0x7f19180ca5a0 msgr2=0x7f19180ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- 172.21.15.136:0/3412500455 shutdown_connections 2024-04-25T11:28:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- 172.21.15.136:0/3412500455 wait complete. 2024-04-25T11:28:25.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 Processor -- start 2024-04-25T11:28:25.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.856+0000 7f191e091640 1 -- start start 2024-04-25T11:28:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f191806c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f191814f840 0x7f191814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1918074d90 con 0x7f1918074980 2024-04-25T11:28:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1918159c10 con 0x7f191814f840 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1918153380 con 0x7f19180c4170 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33826/0 (socket says 172.21.15.136:33826) 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 -- 172.21.15.136:0/3273101218 learned_addr learned my addr 172.21.15.136:0/3273101218 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f1916ffd640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f191806c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f1917fff640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f191814f840 0x7f191814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 -- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 msgr2=0x7f191806c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f191806c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 -- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f191814f840 msgr2=0x7f191814ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f191814f840 0x7f191814ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 -- 172.21.15.136:0/3273101218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191814def0 con 0x7f1918074980 2024-04-25T11:28:25.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f19177fe640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f190800c3a0 tx=0x7f190800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19080052d0 con 0x7f1918074980 2024-04-25T11:28:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1908005470 con 0x7f1918074980 2024-04-25T11:28:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191815d110 con 0x7f1918074980 2024-04-25T11:28:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908007260 con 0x7f1918074980 2024-04-25T11:28:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1918150350 con 0x7f1918074980 2024-04-25T11:28:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.860+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f191806a350 con 0x7f1918074980 2024-04-25T11:28:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.864+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1908007400 con 0x7f1918074980 2024-04-25T11:28:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.864+0000 7f18fffff640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 0x7f1904071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.864+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f190808eb80 con 0x7f1918074980 2024-04-25T11:28:25.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.864+0000 7f1916ffd640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 0x7f1904071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:25.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.864+0000 7f1916ffd640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 0x7f1904071940 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19180c4dc0 tx=0x7f190c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:25.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:25.872+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f190805af80 con 0x7f1918074980 2024-04-25T11:28:26.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.088+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19180c2ed0 con 0x7f1918074980 2024-04-25T11:28:26.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f18fffff640 1 -- 172.21.15.136:0/3273101218 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7f190805f280 con 0x7f1918074980 2024-04-25T11:28:26.081 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:26.081 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:26.081 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 msgr2=0x7f1904071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 0x7f1904071940 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19180c4dc0 tx=0x7f190c009190 comp rx=0 tx=0).stop 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 msgr2=0x7f1918151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f190800c3a0 tx=0x7f190800c860 comp rx=0 tx=0).stop 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 shutdown_connections 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f191814f840 0x7f191814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f190406f430 0x7f1904071940 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:26.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f19180c4170 0x7f191806c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:26.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 --2- 172.21.15.136:0/3273101218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1918074980 0x7f1918151b20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:26.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 >> 172.21.15.136:0/3273101218 conn(0x7f19180ca5a0 msgr2=0x7f1918158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:26.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 shutdown_connections 2024-04-25T11:28:26.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:26.092+0000 7f191e091640 1 -- 172.21.15.136:0/3273101218 wait complete. 2024-04-25T11:28:26.105 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:28.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:28.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 Processor -- start 2024-04-25T11:28:28.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 -- start start 2024-04-25T11:28:28.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa46015c070 0x7fa46006d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa46006da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4600b4480 con 0x7fa4600b5810 2024-04-25T11:28:28.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa4600b4a90 con 0x7fa46015c070 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.280+0000 7fa4659a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa46015c990 con 0x7fa46015c480 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45effd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa4600b5810 0x7fa4600b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:54170/0 (socket says 172.21.15.136:54170) 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45effd640 1 -- 172.21.15.136:0/419380136 learned_addr learned my addr 172.21.15.136:0/419380136 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa46006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa457fff640 1 --2- 172.21.15.136:0/419380136 >> v2:172.21.15.157:3300/0 conn(0x7fa46015c070 0x7fa46006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122347932 0 0) 0x7fa4600b4480 con 0x7fa4600b5810 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa44c003530 con 0x7fa4600b5810 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4282608615 0 0) 0x7fa44c003530 con 0x7fa4600b5810 2024-04-25T11:28:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 msgr2=0x7fa46006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa46006da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 >> v2:172.21.15.157:3300/0 conn(0x7fa46015c070 msgr2=0x7fa46006d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 --2- 172.21.15.136:0/419380136 >> v2:172.21.15.157:3300/0 conn(0x7fa46015c070 0x7fa46006d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46015cc70 con 0x7fa4600b5810 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fa450001680 con 0x7fa4600b5810 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 >> v1:172.21.15.136:6789/0 conn(0x7fa4600b5810 legacy=0x7fa4600b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c0039c0 0x7fa44c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa44c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa44c004d50 0x7fa44c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa4600b4a90 con 0x7fa44c004d50 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa46015c990 con 0x7fa44c0039c0 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa44c00ba80 con 0x7fa46015c480 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa44c004d50 0x7fa44c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45effd640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c0039c0 0x7fa44c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa457fff640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa44c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 -- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 msgr2=0x7fa44c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46015c480 0x7fa44c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 -- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c0039c0 msgr2=0x7fa44c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c0039c0 0x7fa44c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 -- 172.21.15.136:0/419380136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c003530 con 0x7fa44c004d50 2024-04-25T11:28:28.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa44c004d50 0x7fa44c005160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa46008ee40 tx=0x7fa45801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae356d4bb261fa35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:28.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580b2070 con 0x7fa44c004d50 2024-04-25T11:28:28.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4580a5c50 con 0x7fa44c004d50 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.284+0000 7fa45e7fc640 1 -- 172.21.15.136:0/419380136 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458009c10 con 0x7fa44c004d50 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- 172.21.15.136:0/419380136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa44c004d50 msgr2=0x7fa44c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 --2- 172.21.15.136:0/419380136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa44c004d50 0x7fa44c005160 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa46008ee40 tx=0x7fa45801cc40 comp rx=0 tx=0).stop 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa45f7fe640 1 -- 172.21.15.136:0/419380136 reap_dead start 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- 172.21.15.136:0/419380136 shutdown_connections 2024-04-25T11:28:28.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- 172.21.15.136:0/419380136 >> 172.21.15.136:0/419380136 conn(0x7fa4600743a0 msgr2=0x7fa4600d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:28.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- 172.21.15.136:0/419380136 shutdown_connections 2024-04-25T11:28:28.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- 172.21.15.136:0/419380136 wait complete. 2024-04-25T11:28:28.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 Processor -- start 2024-04-25T11:28:28.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.288+0000 7fa4659a2640 1 -- start start 2024-04-25T11:28:28.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 0x7fa4601f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa460156d80 0x7fa4600d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa46006e030 con 0x7fa460155840 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa46015cf50 con 0x7fa44c004d50 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa4600cbee0 con 0x7fa460156d80 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33850/0 (socket says 172.21.15.136:33850) 2024-04-25T11:28:28.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 -- 172.21.15.136:0/1015369594 learned_addr learned my addr 172.21.15.136:0/1015369594 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45effd640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 0x7fa4601f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa457fff640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa460156d80 0x7fa4600d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 -- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa460156d80 msgr2=0x7fa4600d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa460156d80 0x7fa4600d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 -- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 msgr2=0x7fa4601f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 0x7fa4601f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 -- 172.21.15.136:0/1015369594 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa460156190 con 0x7fa460155840 2024-04-25T11:28:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45effd640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 0x7fa4601f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45f7fe640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa458002240 tx=0x7fa458009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580b2040 con 0x7fa460155840 2024-04-25T11:28:28.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4580a5a50 con 0x7fa460155840 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4601f6240 con 0x7fa460155840 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458009c10 con 0x7fa460155840 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.292+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4601f67e0 con 0x7fa460155840 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa4580be020 con 0x7fa460155840 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa45cff9640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 0x7fa440071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:28.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46006a350 con 0x7fa460155840 2024-04-25T11:28:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7fa458002040 con 0x7fa460155840 2024-04-25T11:28:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa45effd640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 0x7fa440071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.296+0000 7fa45effd640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 0x7fa440071940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa450000c10 tx=0x7fa450009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.300+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa45810c080 con 0x7fa460155840 2024-04-25T11:28:28.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.544+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4600cd5f0 con 0x7fa460155840 2024-04-25T11:28:28.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa45cff9640 1 -- 172.21.15.136:0/1015369594 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7fa4580c5020 con 0x7fa460155840 2024-04-25T11:28:28.537 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:28.537 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:28.538 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:28.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 msgr2=0x7fa440071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 0x7fa440071940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa450000c10 tx=0x7fa450009040 comp rx=0 tx=0).stop 2024-04-25T11:28:28.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 msgr2=0x7fa460155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:28.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa458002240 tx=0x7fa458009750 comp rx=0 tx=0).stop 2024-04-25T11:28:28.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 shutdown_connections 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa460155840 0x7fa460155c50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa44006f430 0x7fa440071940 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa460156d80 0x7fa4600d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 --2- 172.21.15.136:0/1015369594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa44c004d50 0x7fa4601f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 >> 172.21.15.136:0/1015369594 conn(0x7fa4600743a0 msgr2=0x7fa46015b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 shutdown_connections 2024-04-25T11:28:28.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:28.548+0000 7fa4659a2640 1 -- 172.21.15.136:0/1015369594 wait complete. 2024-04-25T11:28:28.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:30.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:30.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 Processor -- start 2024-04-25T11:28:30.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 -- start start 2024-04-25T11:28:30.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2b00ca290 0x7fd2b00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 0x7fd2b015a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b015d750 con 0x7fd2b00b5060 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd2b015da30 con 0x7fd2b00ca290 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b7c1f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd2b015dd10 con 0x7fd2b015a430 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2b00ca290 0x7fd2b00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2b00ca290 0x7fd2b00ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33858/0 (socket says 172.21.15.136:33858) 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 -- 172.21.15.136:0/2410144117 learned_addr learned my addr 172.21.15.136:0/2410144117 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b59bb640 1 --1- 172.21.15.136:0/2410144117 >> v1:172.21.15.136:6789/0 conn(0x7fd2b00b5060 0x7fd2b00c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:54180/0 (socket says 172.21.15.136:54180) 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b61bc640 1 --2- 172.21.15.136:0/2410144117 >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 0x7fd2b015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 -- 172.21.15.136:0/2410144117 >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 msgr2=0x7fd2b015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 --2- 172.21.15.136:0/2410144117 >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 0x7fd2b015a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 -- 172.21.15.136:0/2410144117 >> v1:172.21.15.136:6789/0 conn(0x7fd2b00b5060 legacy=0x7fd2b00c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b51ba640 1 -- 172.21.15.136:0/2410144117 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b015dff0 con 0x7fd2b00ca290 2024-04-25T11:28:30.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.728+0000 7fd2b61bc640 1 --2- 172.21.15.136:0/2410144117 >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 0x7fd2b015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:30.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b51ba640 1 --2- 172.21.15.136:0/2410144117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b00ca6a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00b4fe0 tx=0x7fd2a001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399923b0838860e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:30.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b49b9640 1 -- 172.21.15.136:0/2410144117 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00b5070 con 0x7fd2b00ca290 2024-04-25T11:28:30.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b49b9640 1 -- 172.21.15.136:0/2410144117 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a00a8a10 con 0x7fd2b00ca290 2024-04-25T11:28:30.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b49b9640 1 -- 172.21.15.136:0/2410144117 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00a8de0 con 0x7fd2b00ca290 2024-04-25T11:28:30.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/2410144117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 msgr2=0x7fd2b00ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/2410144117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b00ca6a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd2b00b4fe0 tx=0x7fd2a001ed10 comp rx=0 tx=0).stop 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/2410144117 shutdown_connections 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/2410144117 >> v2:172.21.15.157:3300/0 conn(0x7fd2b015a430 0x7fd2b015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/2410144117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b00ca6a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/2410144117 >> 172.21.15.136:0/2410144117 conn(0x7fd2b0070f70 msgr2=0x7fd2b0068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/2410144117 shutdown_connections 2024-04-25T11:28:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.732+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/2410144117 wait complete. 2024-04-25T11:28:30.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 Processor -- start 2024-04-25T11:28:30.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- start start 2024-04-25T11:28:30.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 0x7fd2b00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2b015a430 0x7fd2b01f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b015ad80 con 0x7fd2b00ca290 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b0159f10 con 0x7fd2b015a430 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b015d3f0 con 0x7fd2b00b5060 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33864/0 (socket says 172.21.15.136:33864) 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 -- 172.21.15.136:0/1660029788 learned_addr learned my addr 172.21.15.136:0/1660029788 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b59bb640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 0x7fd2b00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b61bc640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2b015a430 0x7fd2b01f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 -- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 msgr2=0x7fd2b00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 0x7fd2b00b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 -- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2b015a430 msgr2=0x7fd2b01f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2b015a430 0x7fd2b01f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 -- 172.21.15.136:0/1660029788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b00b42e0 con 0x7fd2b00ca290 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b59bb640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 0x7fd2b00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b51ba640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00b3040 tx=0x7fd2a001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00bf070 con 0x7fd2b00ca290 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a00b0840 con 0x7fd2b00ca290 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b00ceaa0 con 0x7fd2b00ca290 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00b5040 con 0x7fd2b00ca290 2024-04-25T11:28:30.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b00b35a0 con 0x7fd2b00ca290 2024-04-25T11:28:30.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.736+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b006a350 con 0x7fd2b00ca290 2024-04-25T11:28:30.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.740+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd2a00c2070 con 0x7fd2b00ca290 2024-04-25T11:28:30.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.740+0000 7fd2a67fc640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 0x7fd294071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:30.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.740+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7fd2a012c080 con 0x7fd2b00ca290 2024-04-25T11:28:30.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.740+0000 7fd2b59bb640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 0x7fd294071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:30.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.740+0000 7fd2b59bb640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 0x7fd294071940 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac007fd0 tx=0x7fd2ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:30.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:30.744+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd2a0018870 con 0x7fd2b00ca290 2024-04-25T11:28:30.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2b00cadd0 con 0x7fd2b00ca290 2024-04-25T11:28:30.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2a67fc640 1 -- 172.21.15.136:0/1660029788 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7fd2a00f8b50 con 0x7fd2b00ca290 2024-04-25T11:28:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:30.990 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:30.990 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:30.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 msgr2=0x7fd294071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 0x7fd294071940 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac007fd0 tx=0x7fd2ac009040 comp rx=0 tx=0).stop 2024-04-25T11:28:30.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 msgr2=0x7fd2b006e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:30.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00b3040 tx=0x7fd2a001bab0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 shutdown_connections 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2b015a430 0x7fd2b01f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2b00ca290 0x7fd2b006e5d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd29406f430 0x7fd294071940 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac007fd0 tx=0x7fd2ac009040 comp rx=0 tx=0).stop 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 --2- 172.21.15.136:0/1660029788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2b00b5060 0x7fd2b00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 >> 172.21.15.136:0/1660029788 conn(0x7fd2b0070f70 msgr2=0x7fd2b014a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:30.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 shutdown_connections 2024-04-25T11:28:30.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:31.000+0000 7fd2b7c1f640 1 -- 172.21.15.136:0/1660029788 wait complete. 2024-04-25T11:28:31.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:33.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:33.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.180+0000 7f554bf95640 1 Processor -- start 2024-04-25T11:28:33.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 -- start start 2024-04-25T11:28:33.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 0x7f55440c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f55440ca290 0x7f55440ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554415d750 con 0x7f554415a430 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f554415da30 con 0x7f55440ca290 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554bf95640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f554415dd10 con 0x7f55440b5060 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f554415a430 0x7f554415a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:54188/0 (socket says 172.21.15.136:54188) 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 -- 172.21.15.136:0/2239618375 learned_addr learned my addr 172.21.15.136:0/2239618375 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5549d31640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 0x7f55440c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669097760 0 0) 0x7f554415d750 con 0x7f554415a430 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5549530640 1 --2- 172.21.15.136:0/2239618375 >> v2:172.21.15.157:3300/0 conn(0x7f55440ca290 0x7f55440ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5530003530 con 0x7f554415a430 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2429699083 0 0) 0x7f5530003530 con 0x7f554415a430 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 msgr2=0x7f55440c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 0x7f55440c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 >> v2:172.21.15.157:3300/0 conn(0x7f55440ca290 msgr2=0x7f55440ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 --2- 172.21.15.136:0/2239618375 >> v2:172.21.15.157:3300/0 conn(0x7f55440ca290 0x7f55440ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554415dff0 con 0x7f554415a430 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f553c001a00 con 0x7f554415a430 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 >> v1:172.21.15.136:6789/0 conn(0x7f554415a430 legacy=0x7f554415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 0x7f55300039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5530003f40 0x7f5530004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5530004d50 0x7f5530005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f554415da30 con 0x7f5530004d50 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f554415dd10 con 0x7f5530003f40 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f553000bab0 con 0x7f55440b5060 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5530004d50 0x7f5530005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5549530640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5530003f40 0x7f5530004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 -- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 msgr2=0x7f55300039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55440b5060 0x7f55300039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 -- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5530003f40 msgr2=0x7f5530004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5530003f40 0x7f5530004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f554a532640 1 -- 172.21.15.136:0/2239618375 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5530003530 con 0x7f5530004d50 2024-04-25T11:28:33.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.184+0000 7f5549530640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5530003f40 0x7f5530004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554a532640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5530004d50 0x7f5530005160 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f553c002240 tx=0x7f553c01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d14d0baaeed56e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c00a4e0 con 0x7f5530004d50 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553c0ad650 con 0x7f5530004d50 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f5548d2f640 1 -- 172.21.15.136:0/2239618375 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c0ada40 con 0x7f5530004d50 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 -- 172.21.15.136:0/2239618375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5530004d50 msgr2=0x7f5530005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 --2- 172.21.15.136:0/2239618375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5530004d50 0x7f5530005160 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f553c002240 tx=0x7f553c01d340 comp rx=0 tx=0).stop 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554a532640 1 -- 172.21.15.136:0/2239618375 reap_dead start 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 -- 172.21.15.136:0/2239618375 shutdown_connections 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 -- 172.21.15.136:0/2239618375 >> 172.21.15.136:0/2239618375 conn(0x7f5544070f70 msgr2=0x7f5544068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 -- 172.21.15.136:0/2239618375 shutdown_connections 2024-04-25T11:28:33.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.188+0000 7f554bf95640 1 -- 172.21.15.136:0/2239618375 wait complete. 2024-04-25T11:28:33.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 Processor -- start 2024-04-25T11:28:33.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- start start 2024-04-25T11:28:33.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f554414fae0 0x7f55441e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 0x7f554414ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f554415d3f0 con 0x7f554414e260 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5544159f10 con 0x7f55441520e0 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55440c4520 con 0x7f554414fae0 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33876/0 (socket says 172.21.15.136:33876) 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 -- 172.21.15.136:0/697032624 learned_addr learned my addr 172.21.15.136:0/697032624 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554a532640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 0x7f554414ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549530640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f554414fae0 0x7f55441e5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 -- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f554414fae0 msgr2=0x7f55441e5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f554414fae0 0x7f55441e5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 -- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 msgr2=0x7f554414ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 0x7f554414ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 -- 172.21.15.136:0/697032624 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5544152da0 con 0x7f554414e260 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554a532640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 0x7f554414ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f5549d31640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f554000e7f0 tx=0x7f554000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5540017070 con 0x7f554414e260 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f554000c940 con 0x7f554414e260 2024-04-25T11:28:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554000eea0 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55440cf140 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55440ce460 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.192+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554406a350 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.196+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f554001a070 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.196+0000 7f553a7fc640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 0x7f5528071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.196+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6907+0+0 (secure 0 0 0) 0x7f554008f5a0 con 0x7f554414e260 2024-04-25T11:28:33.186 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.196+0000 7f5549530640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 0x7f5528071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:33.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.196+0000 7f5549530640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 0x7f5528071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f55440b2840 tx=0x7f5534008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:33.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.200+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f554005c070 con 0x7f554414e260 2024-04-25T11:28:33.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.424+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f554414dec0 con 0x7f554414e260 2024-04-25T11:28:33.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.424+0000 7f553a7fc640 1 -- 172.21.15.136:0/697032624 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6923 (secure 0 0 0) 0x7f554005b970 con 0x7f554414e260 2024-04-25T11:28:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:33.415 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5385,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:26:19.060363+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":4260,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4260":{"gid":4260,"name":"e","rank":3,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6837/1368493675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1368493675},{"type":"v1","addr":"172.21.15.136:6837","nonce":1368493675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269,4260]},"id":1}]} 2024-04-25T11:28:33.415 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 31 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 msgr2=0x7f5528071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 0x7f5528071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f55440b2840 tx=0x7f5534008040 comp rx=0 tx=0).stop 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 msgr2=0x7f55441e5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f554000e7f0 tx=0x7f554000ecb0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 shutdown_connections 2024-04-25T11:28:33.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55441520e0 0x7f554414ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f552806f400 0x7f5528071910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f554414fae0 0x7f55441e5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 --2- 172.21.15.136:0/697032624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f554414e260 0x7f55441e5210 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 >> 172.21.15.136:0/697032624 conn(0x7f5544070f70 msgr2=0x7f55440c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 shutdown_connections 2024-04-25T11:28:33.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:33.428+0000 7f554bf95640 1 -- 172.21.15.136:0/697032624 wait complete. 2024-04-25T11:28:33.435 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:28:35.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:35.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.600+0000 7f426c21f640 1 Processor -- start 2024-04-25T11:28:35.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.600+0000 7f426c21f640 1 -- start start 2024-04-25T11:28:35.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426c21f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f42640c96e0 0x7f42641540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426c21f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f42640c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426c21f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42640c9af0 con 0x7f42640c79e0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426c21f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f426406cc10 con 0x7f42640c96e0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426c21f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42640cb220 con 0x7f426415aae0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f42697ba640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f42640c79e0 0x7f426415a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53408/0 (socket says 172.21.15.136:53408) 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f42697ba640 1 -- 172.21.15.136:0/3788377546 learned_addr learned my addr 172.21.15.136:0/3788377546 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f42640c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4269fbb640 1 --2- 172.21.15.136:0/3788377546 >> v2:172.21.15.157:3300/0 conn(0x7f42640c96e0 0x7f42641540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3448506673 0 0) 0x7f42640c9af0 con 0x7f42640c79e0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4250003530 con 0x7f42640c79e0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 120325911 0 0) 0x7f4250003530 con 0x7f42640c79e0 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 msgr2=0x7f42640c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f42640c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 >> v2:172.21.15.157:3300/0 conn(0x7f42640c96e0 msgr2=0x7f42641540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 --2- 172.21.15.136:0/3788377546 >> v2:172.21.15.157:3300/0 conn(0x7f42640c96e0 0x7f42641540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42640cb500 con 0x7f42640c79e0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f42540016a0 con 0x7f42640c79e0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 >> v1:172.21.15.136:6789/0 conn(0x7f42640c79e0 legacy=0x7f426415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42500039c0 0x7f4250003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f4250004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4250004d50 0x7f4250005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f426406cc10 con 0x7f4250004d50 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f42640cb220 con 0x7f42500039c0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f425000ba50 con 0x7f426415aae0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4250004d50 0x7f4250005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f4269fbb640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42500039c0 0x7f4250003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f42697ba640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f4250004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 -- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 msgr2=0x7f4250004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f4250004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 -- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42500039c0 msgr2=0x7f4250003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f42500039c0 0x7f4250003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f426a7bc640 1 -- 172.21.15.136:0/3788377546 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4250003530 con 0x7f4250004d50 2024-04-25T11:28:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.604+0000 7f42697ba640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f426415aae0 0x7f4250004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:35.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426a7bc640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4250004d50 0x7f4250005160 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4264075160 tx=0x7f425c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e506cebbafc571c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:35.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0b2070 con 0x7f4250004d50 2024-04-25T11:28:35.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425c0a5c50 con 0x7f4250004d50 2024-04-25T11:28:35.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f4268fb9640 1 -- 172.21.15.136:0/3788377546 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c009c10 con 0x7f4250004d50 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 -- 172.21.15.136:0/3788377546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4250004d50 msgr2=0x7f4250005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 --2- 172.21.15.136:0/3788377546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4250004d50 0x7f4250005160 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4264075160 tx=0x7f425c01cc40 comp rx=0 tx=0).stop 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426a7bc640 1 -- 172.21.15.136:0/3788377546 reap_dead start 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 -- 172.21.15.136:0/3788377546 shutdown_connections 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 -- 172.21.15.136:0/3788377546 >> 172.21.15.136:0/3788377546 conn(0x7f426406f9d0 msgr2=0x7f426406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 -- 172.21.15.136:0/3788377546 shutdown_connections 2024-04-25T11:28:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.608+0000 7f426c21f640 1 -- 172.21.15.136:0/3788377546 wait complete. 2024-04-25T11:28:35.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 Processor -- start 2024-04-25T11:28:35.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- start start 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4264157d00 0x7f4264158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4264149ad0 0x7f4264149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f42640cf0f0 con 0x7f4264141900 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4264154610 con 0x7f4264149ad0 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f42640ced40 con 0x7f4264157d00 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43488/0 (socket says 172.21.15.136:43488) 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 -- 172.21.15.136:0/2618508477 learned_addr learned my addr 172.21.15.136:0/2618508477 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f42697ba640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4264157d00 0x7f4264158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426a7bc640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4264149ad0 0x7f4264149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 -- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4264157d00 msgr2=0x7f4264158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4264157d00 0x7f4264158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 -- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4264149ad0 msgr2=0x7f4264149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4264149ad0 0x7f4264149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 -- 172.21.15.136:0/2618508477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42641549d0 con 0x7f4264141900 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f4269fbb640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42600119d0 tx=0x7f4260011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000fb50 con 0x7f4264141900 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426000fcf0 con 0x7f4264141900 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42641434f0 con 0x7f4264141900 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000fea0 con 0x7f4264141900 2024-04-25T11:28:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.612+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4264147890 con 0x7f4264141900 2024-04-25T11:28:35.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f426406a350 con 0x7f4264141900 2024-04-25T11:28:35.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4260012040 con 0x7f4264141900 2024-04-25T11:28:35.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f425a7fc640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 0x7f4248071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:35.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f426008fe70 con 0x7f4264141900 2024-04-25T11:28:35.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f42697ba640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 0x7f4248071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:35.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.616+0000 7f42697ba640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 0x7f4248071910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c10 tx=0x7f4254009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:35.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.620+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4260019030 con 0x7f4264141900 2024-04-25T11:28:35.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.832+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42640c51f0 con 0x7f4264141900 2024-04-25T11:28:35.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f425a7fc640 1 -- 172.21.15.136:0/2618508477 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6090 (secure 0 0 0) 0x7f426005c9a0 con 0x7f4264141900 2024-04-25T11:28:35.825 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:35.825 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:33.854127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:28:35.825 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 32 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 msgr2=0x7f4248071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 0x7f4248071910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4254000c10 tx=0x7f4254009040 comp rx=0 tx=0).stop 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 msgr2=0x7f42641572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42600119d0 tx=0x7f4260011e90 comp rx=0 tx=0).stop 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 shutdown_connections 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4264149ad0 0x7f4264149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f424806f400 0x7f4248071910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4264157d00 0x7f4264158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 --2- 172.21.15.136:0/2618508477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4264141900 0x7f42641572b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:35.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 >> 172.21.15.136:0/2618508477 conn(0x7f426406f9d0 msgr2=0x7f4264152dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:35.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 shutdown_connections 2024-04-25T11:28:35.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:35.836+0000 7f426c21f640 1 -- 172.21.15.136:0/2618508477 wait complete. 2024-04-25T11:28:35.852 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-04-25T11:28:35.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:28:35.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:36.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 Processor -- start 2024-04-25T11:28:36.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 -- start start 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b0c156750 0x7f0b0c0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b0c0cfba0 0x7f0b0c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b0c15c410 con 0x7f0b0c0ca7d0 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0b0c15c6b0 con 0x7f0b0c156750 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b13ce6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b0c15c990 con 0x7f0b0c0cfba0 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b0c156750 0x7f0b0c0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b0c156750 0x7f0b0c0ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43496/0 (socket says 172.21.15.136:43496) 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 -- 172.21.15.136:0/2325115260 learned_addr learned my addr 172.21.15.136:0/2325115260 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b12283640 1 --2- 172.21.15.136:0/2325115260 >> v2:172.21.15.157:3300/0 conn(0x7f0b0c0cfba0 0x7f0b0c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 -- 172.21.15.136:0/2325115260 >> v2:172.21.15.157:3300/0 conn(0x7f0b0c0cfba0 msgr2=0x7f0b0c06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:36.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 --2- 172.21.15.136:0/2325115260 >> v2:172.21.15.157:3300/0 conn(0x7f0b0c0cfba0 0x7f0b0c06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 -- 172.21.15.136:0/2325115260 >> v1:172.21.15.157:6789/0 conn(0x7f0b0c0ca7d0 legacy=0x7f0b0c0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:36.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.016+0000 7f0b11a82640 1 -- 172.21.15.136:0/2325115260 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0c15cc70 con 0x7f0b0c156750 2024-04-25T11:28:36.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b11a82640 1 --2- 172.21.15.136:0/2325115260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c156750 0x7f0b0c0ca290 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0b5850 tx=0x7f0b0801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4915c90e462f999c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:36.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b10a80640 1 -- 172.21.15.136:0/2325115260 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b080b0070 con 0x7f0b0c156750 2024-04-25T11:28:36.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b10a80640 1 -- 172.21.15.136:0/2325115260 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b080ab600 con 0x7f0b0c156750 2024-04-25T11:28:36.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b10a80640 1 -- 172.21.15.136:0/2325115260 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b080ab9d0 con 0x7f0b0c156750 2024-04-25T11:28:36.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 -- 172.21.15.136:0/2325115260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c156750 msgr2=0x7f0b0c0ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:36.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/2325115260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c156750 0x7f0b0c0ca290 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0b5850 tx=0x7f0b0801c4e0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 -- 172.21.15.136:0/2325115260 shutdown_connections 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/2325115260 >> v2:172.21.15.157:3300/0 conn(0x7f0b0c0cfba0 0x7f0b0c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/2325115260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c156750 0x7f0b0c0ca290 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 -- 172.21.15.136:0/2325115260 >> 172.21.15.136:0/2325115260 conn(0x7f0b0c070f70 msgr2=0x7f0b0c068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 -- 172.21.15.136:0/2325115260 shutdown_connections 2024-04-25T11:28:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.020+0000 7f0b13ce6640 1 -- 172.21.15.136:0/2325115260 wait complete. 2024-04-25T11:28:36.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 Processor -- start 2024-04-25T11:28:36.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- start start 2024-04-25T11:28:36.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0c0ca7d0 0x7f0b0c1f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 0x7f0b0c15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1692c0 con 0x7f0b0c14f440 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c06da70 con 0x7f0b0c0cfba0 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b0c169440 con 0x7f0b0c0ca7d0 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43500/0 (socket says 172.21.15.136:43500) 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 -- 172.21.15.136:0/3423977881 learned_addr learned my addr 172.21.15.136:0/3423977881 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b11281640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 0x7f0b0c15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b11a82640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0c0ca7d0 0x7f0b0c1f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 -- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0c0ca7d0 msgr2=0x7f0b0c1f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0c0ca7d0 0x7f0b0c1f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 -- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 msgr2=0x7f0b0c15a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 0x7f0b0c15a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 -- 172.21.15.136:0/3423977881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0c1f93d0 con 0x7f0b0c14f440 2024-04-25T11:28:36.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b11281640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 0x7f0b0c15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b12283640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009fd0 tx=0x7f0b0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04014420 con 0x7f0b0c14f440 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0c1f53b0 con 0x7f0b0c14f440 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b04019070 con 0x7f0b0c14f440 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04014800 con 0x7f0b0c14f440 2024-04-25T11:28:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b0c1f9690 con 0x7f0b0c14f440 2024-04-25T11:28:36.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.024+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b0c1528b0 con 0x7f0b0c14f440 2024-04-25T11:28:36.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.028+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0b040149a0 con 0x7f0b0c14f440 2024-04-25T11:28:36.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.028+0000 7f0b027fc640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 0x7f0af0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:36.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.028+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f0b0408ffc0 con 0x7f0b0c14f440 2024-04-25T11:28:36.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.028+0000 7f0b11a82640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 0x7f0af0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:36.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.028+0000 7f0b11a82640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 0x7f0af0071910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0b080ad000 tx=0x7f0b080a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:36.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.032+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0b0405c920 con 0x7f0b0c14f440 2024-04-25T11:28:36.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.288+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b0c06a350 con 0x7f0b0c14f440 2024-04-25T11:28:36.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b027fc640 1 -- 172.21.15.136:0/3423977881 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6090 (secure 0 0 0) 0x7f0b0405c220 con 0x7f0b0c14f440 2024-04-25T11:28:36.281 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:36.281 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:33.854127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:28:36.281 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 32 2024-04-25T11:28:36.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 msgr2=0x7f0af0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:36.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 0x7f0af0071910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0b080ad000 tx=0x7f0b080a6000 comp rx=0 tx=0).stop 2024-04-25T11:28:36.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 msgr2=0x7f0b0c14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009fd0 tx=0x7f0b0400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 shutdown_connections 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c14f440 0x7f0b0c14f850 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0c0cfba0 0x7f0b0c15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0af006f400 0x7f0af0071910 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 --2- 172.21.15.136:0/3423977881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0c0ca7d0 0x7f0b0c1f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 >> 172.21.15.136:0/3423977881 conn(0x7f0b0c070f70 msgr2=0x7f0b0c0cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 shutdown_connections 2024-04-25T11:28:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:36.292+0000 7f0b13ce6640 1 -- 172.21.15.136:0/3423977881 wait complete. 2024-04-25T11:28:36.308 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:28:33.854127+0000", "qdb_cluster": [ 4269, 4297, 4303, 5238 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:28:38.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.488+0000 7f6eb1ac3640 1 Processor -- start 2024-04-25T11:28:38.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.488+0000 7f6eb1ac3640 1 -- start start 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6eac153d60 0x7f6eac15b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6eac15bb60 0x7f6eac15d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f6eac0703c0 con 0x7f6eac153d60 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6eac070680 con 0x7f6eac15bb60 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eb1ac3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6eac070960 con 0x7f6eac15d8a0 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6eac153d60 0x7f6eac15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6eac153d60 0x7f6eac15b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43514/0 (socket says 172.21.15.136:43514) 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 -- 172.21.15.136:0/1482895868 learned_addr learned my addr 172.21.15.136:0/1482895868 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eabfff640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eaaffd640 1 --2- 172.21.15.136:0/1482895868 >> v2:172.21.15.157:3300/0 conn(0x7f6eac15bb60 0x7f6eac15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 -- 172.21.15.136:0/1482895868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 msgr2=0x7f6eac15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 -- 172.21.15.136:0/1482895868 >> v2:172.21.15.157:3300/0 conn(0x7f6eac15bb60 msgr2=0x7f6eac15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1482895868 >> v2:172.21.15.157:3300/0 conn(0x7f6eac15bb60 0x7f6eac15d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.492+0000 7f6eab7fe640 1 -- 172.21.15.136:0/1482895868 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac070c40 con 0x7f6eac153d60 2024-04-25T11:28:38.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac153d60 0x7f6eac15b620 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6eac075040 tx=0x7f6e9c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b85ca69eb432bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:38.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eaa7fc640 1 -- 172.21.15.136:0/1482895868 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0ab5f0 con 0x7f6eac153d60 2024-04-25T11:28:38.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eaa7fc640 1 -- 172.21.15.136:0/1482895868 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9c0b0070 con 0x7f6eac153d60 2024-04-25T11:28:38.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eaa7fc640 1 -- 172.21.15.136:0/1482895868 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0ab9b0 con 0x7f6eac153d60 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1482895868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac153d60 msgr2=0x7f6eac15b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac153d60 0x7f6eac15b620 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6eac075040 tx=0x7f6e9c01c520 comp rx=0 tx=0).stop 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1482895868 shutdown_connections 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1482895868 >> v2:172.21.15.157:3300/0 conn(0x7f6eac15bb60 0x7f6eac15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1482895868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac153d60 0x7f6eac15b620 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1482895868 >> 172.21.15.136:0/1482895868 conn(0x7f6eac0cd4a0 msgr2=0x7f6eac0cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1482895868 shutdown_connections 2024-04-25T11:28:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.496+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1482895868 wait complete. 2024-04-25T11:28:38.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.500+0000 7f6eb1ac3640 1 Processor -- start 2024-04-25T11:28:38.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.500+0000 7f6eb1ac3640 1 -- start start 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 0x7f6eac1f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac0c63f0 con 0x7f6eac15bb60 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac0c76a0 con 0x7f6eac153d60 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6eac154170 con 0x7f6eac15d8a0 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43518/0 (socket says 172.21.15.136:43518) 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 -- 172.21.15.136:0/1739815916 learned_addr learned my addr 172.21.15.136:0/1739815916 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 0x7f6eac1f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 -- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 msgr2=0x7f6eac14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eabfff640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac14c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 -- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 msgr2=0x7f6eac1f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 0x7f6eac1f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 -- 172.21.15.136:0/1739815916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac1f3760 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 0x7f6eac1f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eaaffd640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000d5f0 tx=0x7f6ea000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00125f0 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea0019070 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac1f4430 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00129b0 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac0ccf80 con 0x7f6eac15bb60 2024-04-25T11:28:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.504+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac06a350 con 0x7f6eac15bb60 2024-04-25T11:28:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.508+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6ea0012b50 con 0x7f6eac15bb60 2024-04-25T11:28:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.508+0000 7f6e93fff640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 0x7f6e98071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.508+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f6ea008fbf0 con 0x7f6eac15bb60 2024-04-25T11:28:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.508+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 0x7f6e98071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.508+0000 7f6eab7fe640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 0x7f6e98071910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0af000 tx=0x7f6e9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:38.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.512+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6ea005c5d0 con 0x7f6eac15bb60 2024-04-25T11:28:38.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eac0c5190 con 0x7f6eac15bb60 2024-04-25T11:28:38.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6e93fff640 1 -- 172.21.15.136:0/1739815916 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6090 (secure 0 0 0) 0x7f6ea005bed0 con 0x7f6eac15bb60 2024-04-25T11:28:38.734 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:38.734 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:33.854127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:28:38.734 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 32 2024-04-25T11:28:38.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 msgr2=0x7f6e98071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:38.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 0x7f6e98071910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c0af000 tx=0x7f6e9c008040 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 msgr2=0x7f6eac14b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000d5f0 tx=0x7f6ea000dab0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 shutdown_connections 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6e9806f400 0x7f6e98071910 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6eac15d8a0 0x7f6eac14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eac15bb60 0x7f6eac14b720 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 --2- 172.21.15.136:0/1739815916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6eac153d60 0x7f6eac1f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.744+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 >> 172.21.15.136:0/1739815916 conn(0x7f6eac0cd4a0 msgr2=0x7f6eac151430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.748+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 shutdown_connections 2024-04-25T11:28:38.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:38.748+0000 7f6eb1ac3640 1 -- 172.21.15.136:0/1739815916 wait complete. 2024-04-25T11:28:38.759 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:40.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:40.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.924+0000 7f06dcd33640 1 Processor -- start 2024-04-25T11:28:40.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.924+0000 7f06dcd33640 1 -- start start 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06dcd33640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f06d815a5f0 0x7f06d80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06dcd33640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06d80c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06dcd33640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d80c3880 con 0x7f06d8154060 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06dcd33640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f06d80c8fd0 con 0x7f06d815a5f0 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06dcd33640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06d806d9b0 con 0x7f06d815aa00 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f06d8154060 0x7f06d80c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53422/0 (socket says 172.21.15.136:53422) 2024-04-25T11:28:40.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d659c640 1 -- 172.21.15.136:0/3364878137 learned_addr learned my addr 172.21.15.136:0/3364878137 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:40.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d5d9b640 1 --2- 172.21.15.136:0/3364878137 >> v2:172.21.15.157:3300/0 conn(0x7f06d815a5f0 0x7f06d80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06d80c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550495360 0 0) 0x7f06d80c3880 con 0x7f06d8154060 2024-04-25T11:28:40.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c0003530 con 0x7f06d8154060 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1237032638 0 0) 0x7f06c0003530 con 0x7f06d8154060 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 msgr2=0x7f06d80c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06d80c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 >> v2:172.21.15.157:3300/0 conn(0x7f06d815a5f0 msgr2=0x7f06d80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 --2- 172.21.15.136:0/3364878137 >> v2:172.21.15.157:3300/0 conn(0x7f06d815a5f0 0x7f06d80c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d8154470 con 0x7f06d8154060 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f06c8001680 con 0x7f06d8154060 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 >> v1:172.21.15.136:6789/0 conn(0x7f06d8154060 legacy=0x7f06d80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00039c0 0x7f06c0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c0004d50 0x7f06c0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f06d80c8fd0 con 0x7f06c0004d50 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06d806d9b0 con 0x7f06c00039c0 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06c000ba80 con 0x7f06d815aa00 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c0004d50 0x7f06c0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d659c640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00039c0 0x7f06c0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d5d9b640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 msgr2=0x7f06c0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06c0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00039c0 msgr2=0x7f06c0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00039c0 0x7f06c0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3364878137 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c0003530 con 0x7f06c0004d50 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d5d9b640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815aa00 0x7f06c0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.928+0000 7f06d659c640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06c00039c0 0x7f06c0003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:40.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c0004d50 0x7f06c0005160 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009f90 tx=0x7f06d001ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830748ef1a6fb408 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:40.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00b2070 con 0x7f06c0004d50 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d00a5d00 con 0x7f06c0004d50 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06d559a640 1 -- 172.21.15.136:0/3364878137 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0009c10 con 0x7f06c0004d50 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 -- 172.21.15.136:0/3364878137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c0004d50 msgr2=0x7f06c0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3364878137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06c0004d50 0x7f06c0005160 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009f90 tx=0x7f06d001ccf0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3364878137 reap_dead start 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 -- 172.21.15.136:0/3364878137 shutdown_connections 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 -- 172.21.15.136:0/3364878137 >> 172.21.15.136:0/3364878137 conn(0x7f06d80ca5a0 msgr2=0x7f06d80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 -- 172.21.15.136:0/3364878137 shutdown_connections 2024-04-25T11:28:40.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.932+0000 7f06dcd33640 1 -- 172.21.15.136:0/3364878137 wait complete. 2024-04-25T11:28:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 Processor -- start 2024-04-25T11:28:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 -- start start 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d814a5d0 0x7f06d80cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 0x7f06d81e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f06d815a0d0 con 0x7f06d81e5270 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06d806df30 con 0x7f06d81f0ad0 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06dcd33640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06d8159d80 con 0x7f06d814a5d0 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43532/0 (socket says 172.21.15.136:43532) 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3410402010 learned_addr learned my addr 172.21.15.136:0/3410402010 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d659c640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d814a5d0 0x7f06d80cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d5d9b640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 0x7f06d81e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d814a5d0 msgr2=0x7f06d80cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d814a5d0 0x7f06d80cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 msgr2=0x7f06d81e4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 0x7f06d81e4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 -- 172.21.15.136:0/3410402010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d8153090 con 0x7f06d81e5270 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d5d9b640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 0x7f06d81e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.936+0000 7f06d6d9d640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009f90 tx=0x7f06d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00b2040 con 0x7f06d81e5270 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d00039a0 con 0x7f06d81e5270 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d8153370 con 0x7f06d81e5270 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0009c10 con 0x7f06d81e5270 2024-04-25T11:28:40.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d81e5680 con 0x7f06d81e5270 2024-04-25T11:28:40.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d806a350 con 0x7f06d81e5270 2024-04-25T11:28:40.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f06d00be060 con 0x7f06d81e5270 2024-04-25T11:28:40.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 0x7f06b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:40.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f06d0129110 con 0x7f06d81e5270 2024-04-25T11:28:40.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06d659c640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 0x7f06b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:40.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.940+0000 7f06d659c640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 0x7f06b8071910 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f06c80018f0 tx=0x7f06c8001860 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:40.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:40.948+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f06d0016970 con 0x7f06d81e5270 2024-04-25T11:28:41.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.156+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06d80c85e0 con 0x7f06d81e5270 2024-04-25T11:28:41.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06c6ffd640 1 -- 172.21.15.136:0/3410402010 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6092 (secure 0 0 0) 0x7f06d00f5c10 con 0x7f06d81e5270 2024-04-25T11:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:38.868543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:resolve","state_seq":34,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:28:41.149 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 33 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 msgr2=0x7f06b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 0x7f06b8071910 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f06c80018f0 tx=0x7f06c8001860 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 msgr2=0x7f06d8145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009f90 tx=0x7f06d0009750 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 shutdown_connections 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d81e5270 0x7f06d8145e10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d81f0ad0 0x7f06d81e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f06b806f400 0x7f06b8071910 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 --2- 172.21.15.136:0/3410402010 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d814a5d0 0x7f06d80cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 >> 172.21.15.136:0/3410402010 conn(0x7f06d80ca5a0 msgr2=0x7f06d80c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 shutdown_connections 2024-04-25T11:28:41.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:41.160+0000 7f06dcd33640 1 -- 172.21.15.136:0/3410402010 wait complete. 2024-04-25T11:28:41.177 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:43.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:43.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.356+0000 7f1be4912640 1 Processor -- start 2024-04-25T11:28:43.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.356+0000 7f1be4912640 1 -- start start 2024-04-25T11:28:43.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1be4912640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 0x7f1be00c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1be4912640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1be015ab40 0x7f1be00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1be4912640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be015d750 con 0x7f1be00ca290 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1be4912640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1be015da30 con 0x7f1be015ab40 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1be4912640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1be015dd10 con 0x7f1be00b5060 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1be015ab40 0x7f1be00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1be015ab40 0x7f1be00ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43548/0 (socket says 172.21.15.136:43548) 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 -- 172.21.15.136:0/3999894118 learned_addr learned my addr 172.21.15.136:0/3999894118 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bd7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1be00ca290 0x7f1be015a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53434/0 (socket says 172.21.15.136:53434) 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bde59c640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 0x7f1be00c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 -- 172.21.15.136:0/3999894118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 msgr2=0x7f1be00c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 0x7f1be00c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 -- 172.21.15.136:0/3999894118 >> v1:172.21.15.136:6789/0 conn(0x7f1be00ca290 legacy=0x7f1be015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bded9d640 1 -- 172.21.15.136:0/3999894118 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be015dff0 con 0x7f1be015ab40 2024-04-25T11:28:43.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.360+0000 7f1bde59c640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 0x7f1be00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:43.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1bded9d640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be015ab40 0x7f1be00ca6a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009e50 tx=0x7f1bd801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85849814f2290992 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:43.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1bddd9b640 1 -- 172.21.15.136:0/3999894118 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80b4070 con 0x7f1be015ab40 2024-04-25T11:28:43.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1bddd9b640 1 -- 172.21.15.136:0/3999894118 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd80af400 con 0x7f1be015ab40 2024-04-25T11:28:43.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1bddd9b640 1 -- 172.21.15.136:0/3999894118 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80af7d0 con 0x7f1be015ab40 2024-04-25T11:28:43.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 -- 172.21.15.136:0/3999894118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be015ab40 msgr2=0x7f1be00ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be015ab40 0x7f1be00ca6a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009e50 tx=0x7f1bd801f210 comp rx=0 tx=0).stop 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 -- 172.21.15.136:0/3999894118 shutdown_connections 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be015ab40 0x7f1be00ca6a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 --2- 172.21.15.136:0/3999894118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be00b5060 0x7f1be00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 -- 172.21.15.136:0/3999894118 >> 172.21.15.136:0/3999894118 conn(0x7f1be0070f70 msgr2=0x7f1be0068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 -- 172.21.15.136:0/3999894118 shutdown_connections 2024-04-25T11:28:43.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.364+0000 7f1be4912640 1 -- 172.21.15.136:0/3999894118 wait complete. 2024-04-25T11:28:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 Processor -- start 2024-04-25T11:28:43.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- start start 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be00ca290 0x7f1be00cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be015ab40 0x7f1be0147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1be015d3f0 con 0x7f1be00b5060 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0159f10 con 0x7f1be00ca290 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1be00c4a60 con 0x7f1be015ab40 2024-04-25T11:28:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43556/0 (socket says 172.21.15.136:43556) 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 -- 172.21.15.136:0/3243972251 learned_addr learned my addr 172.21.15.136:0/3243972251 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bded9d640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be015ab40 0x7f1be0147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bd7fff640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be00ca290 0x7f1be00cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 -- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be015ab40 msgr2=0x7f1be0147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be015ab40 0x7f1be0147070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 -- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be00ca290 msgr2=0x7f1be00cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be00ca290 0x7f1be00cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 -- 172.21.15.136:0/3243972251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be01f15d0 con 0x7f1be00b5060 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bde59c640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d800 tx=0x7f1bd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0018020 con 0x7f1be00b5060 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd0009660 con 0x7f1be00b5060 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be01f08d0 con 0x7f1be00b5060 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0009a50 con 0x7f1be00b5060 2024-04-25T11:28:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.368+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be014ffa0 con 0x7f1be00b5060 2024-04-25T11:28:43.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be01f06c0 con 0x7f1be00b5060 2024-04-25T11:28:43.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1bd001b020 con 0x7f1be00b5060 2024-04-25T11:28:43.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1bd6ffd640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:43.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f1bd008fc70 con 0x7f1be00b5060 2024-04-25T11:28:43.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1bd7fff640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:43.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.372+0000 7f1bd7fff640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 0x7f1bc0071910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1be00ca8c0 tx=0x7f1bc8007cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:43.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.380+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1bd0022020 con 0x7f1be00b5060 2024-04-25T11:28:43.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.604+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1be006a350 con 0x7f1be00b5060 2024-04-25T11:28:43.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.604+0000 7f1bd6ffd640 1 -- 172.21.15.136:0/3243972251 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6092 (secure 0 0 0) 0x7f1bd005c6e0 con 0x7f1be00b5060 2024-04-25T11:28:43.594 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:43.595 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:38.868543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:resolve","state_seq":34,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:28:43.595 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 33 2024-04-25T11:28:43.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 msgr2=0x7f1bc0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 0x7f1bc0071910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1be00ca8c0 tx=0x7f1bc8007cf0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 msgr2=0x7f1be01f1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d800 tx=0x7f1bd000dcc0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 shutdown_connections 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1be015ab40 0x7f1be0147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1be00ca290 0x7f1be00cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 --2- 172.21.15.136:0/3243972251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be00b5060 0x7f1be01f1090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 >> 172.21.15.136:0/3243972251 conn(0x7f1be0070f70 msgr2=0x7f1be0150490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 shutdown_connections 2024-04-25T11:28:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:43.608+0000 7f1be4912640 1 -- 172.21.15.136:0/3243972251 wait complete. 2024-04-25T11:28:43.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:45.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:45.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 Processor -- start 2024-04-25T11:28:45.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 -- start start 2024-04-25T11:28:45.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 0x7f57740c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f577415aa00 0x7f5774164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740c8d40 con 0x7f577415a5f0 2024-04-25T11:28:45.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f57740c8fd0 con 0x7f577415aa00 2024-04-25T11:28:45.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577a2d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f57740c3880 con 0x7f5774154060 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f57737fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f577415a5f0 0x7f57740c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:58420/0 (socket says 172.21.15.136:58420) 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f57737fe640 1 -- 172.21.15.136:0/2986975782 learned_addr learned my addr 172.21.15.136:0/2986975782 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5773fff640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 0x7f57740c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f577886d640 1 --2- 172.21.15.136:0/2986975782 >> v2:172.21.15.157:3300/0 conn(0x7f577415aa00 0x7f5774164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139305119 0 0) 0x7f57740c8d40 con 0x7f577415a5f0 2024-04-25T11:28:45.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f575c003530 con 0x7f577415a5f0 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3276840740 0 0) 0x7f575c003530 con 0x7f577415a5f0 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 msgr2=0x7f57740c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 0x7f57740c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 >> v2:172.21.15.157:3300/0 conn(0x7f577415aa00 msgr2=0x7f5774164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 --2- 172.21.15.136:0/2986975782 >> v2:172.21.15.157:3300/0 conn(0x7f577415aa00 0x7f5774164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577406d9b0 con 0x7f577415a5f0 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f5768001680 con 0x7f577415a5f0 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 >> v1:172.21.15.136:6789/0 conn(0x7f577415a5f0 legacy=0x7f57740c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f575c0039c0 0x7f575c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 0x7f575c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f575c004d50 0x7f575c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f57740c8fd0 con 0x7f575c0039c0 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f57740c3880 con 0x7f575c004d50 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f575c00ba80 con 0x7f5774154060 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.796+0000 7f5773fff640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f575c0039c0 0x7f575c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577886d640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f575c004d50 0x7f575c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 -- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 msgr2=0x7f575c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5774154060 0x7f575c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 -- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f575c004d50 msgr2=0x7f575c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f575c004d50 0x7f575c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 -- 172.21.15.136:0/2986975782 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c003530 con 0x7f575c0039c0 2024-04-25T11:28:45.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577886d640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f575c004d50 0x7f575c005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5773fff640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f575c0039c0 0x7f575c003e50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5764009ae0 tx=0x7f576401ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a6d7c8cca84f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640b2070 con 0x7f575c0039c0 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57640a5670 con 0x7f575c0039c0 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f5772ffd640 1 -- 172.21.15.136:0/2986975782 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640096b0 con 0x7f575c0039c0 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577a2d0640 1 -- 172.21.15.136:0/2986975782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f575c0039c0 msgr2=0x7f575c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577a2d0640 1 --2- 172.21.15.136:0/2986975782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f575c0039c0 0x7f575c003e50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5764009ae0 tx=0x7f576401ca80 comp rx=0 tx=0).stop 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577886d640 1 -- 172.21.15.136:0/2986975782 reap_dead start 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577a2d0640 1 -- 172.21.15.136:0/2986975782 shutdown_connections 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.800+0000 7f577a2d0640 1 -- 172.21.15.136:0/2986975782 >> 172.21.15.136:0/2986975782 conn(0x7f57740ca5a0 msgr2=0x7f57740ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- 172.21.15.136:0/2986975782 shutdown_connections 2024-04-25T11:28:45.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- 172.21.15.136:0/2986975782 wait complete. 2024-04-25T11:28:45.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 Processor -- start 2024-04-25T11:28:45.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- start start 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f575c0039c0 0x7f57741f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 0x7f577414b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f577406df30 con 0x7f5774152990 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5774154470 con 0x7f57740cd100 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577a2d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f577415a0d0 con 0x7f575c0039c0 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f57737fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f57737fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33230/0 (socket says 172.21.15.136:33230) 2024-04-25T11:28:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f57737fe640 1 -- 172.21.15.136:0/4037388157 learned_addr learned my addr 172.21.15.136:0/4037388157 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f5773fff640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f575c0039c0 0x7f57741f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.804+0000 7f577886d640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 0x7f577414b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 -- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f575c0039c0 msgr2=0x7f57741f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f575c0039c0 0x7f57741f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 -- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 msgr2=0x7f577414b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 0x7f577414b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 -- 172.21.15.136:0/4037388157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577414e820 con 0x7f5774152990 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f577886d640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 0x7f577414b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f57737fe640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f576800f3a0 tx=0x7f576800f860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768007d00 con 0x7f5774152990 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576800a0d0 con 0x7f5774152990 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576800a4a0 con 0x7f5774152990 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57741576a0 con 0x7f5774152990 2024-04-25T11:28:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5774156770 con 0x7f5774152990 2024-04-25T11:28:45.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577406a350 con 0x7f5774152990 2024-04-25T11:28:45.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5768014070 con 0x7f5774152990 2024-04-25T11:28:45.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 0x7f5760071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:45.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f5768092930 con 0x7f5774152990 2024-04-25T11:28:45.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.808+0000 7f5773fff640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 0x7f5760071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:45.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.812+0000 7f5773fff640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 0x7f5760071910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f57640046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:45.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:45.816+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f576805f3f0 con 0x7f5774152990 2024-04-25T11:28:46.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57740c85e0 con 0x7f5774152990 2024-04-25T11:28:46.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f5770ff9640 1 -- 172.21.15.136:0/4037388157 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6094 (secure 0 0 0) 0x7f576805ecf0 con 0x7f5774152990 2024-04-25T11:28:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:46.054 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:44.143342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:28:46.054 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 34 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 msgr2=0x7f5760071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 0x7f5760071910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f57640046d0 comp rx=0 tx=0).stop 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 msgr2=0x7f577414e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f576800f3a0 tx=0x7f576800f860 comp rx=0 tx=0).stop 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 shutdown_connections 2024-04-25T11:28:46.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57740cd100 0x7f577414b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5774152990 0x7f577414e2e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f576006f400 0x7f5760071910 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 --2- 172.21.15.136:0/4037388157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f575c0039c0 0x7f57741f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 >> 172.21.15.136:0/4037388157 conn(0x7f57740ca5a0 msgr2=0x7f57740c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 shutdown_connections 2024-04-25T11:28:46.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:46.064+0000 7f577a2d0640 1 -- 172.21.15.136:0/4037388157 wait complete. 2024-04-25T11:28:46.079 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:48.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:48.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.264+0000 7f342f618640 1 Processor -- start 2024-04-25T11:28:48.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.264+0000 7f342f618640 1 -- start start 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342f618640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342f618640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f342815ab40 0x7f34280ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342f618640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f342815d750 con 0x7f34280ca290 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342f618640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f342815da30 con 0x7f342815ab40 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342f618640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f342815dd10 con 0x7f34280b5060 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342dbb5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f342815ab40 0x7f34280ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:37614/0 (socket says 172.21.15.136:37614) 2024-04-25T11:28:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 -- 172.21.15.136:0/1030820718 learned_addr learned my addr 172.21.15.136:0/1030820718 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:48.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 -- 172.21.15.136:0/1030820718 >> v2:172.21.15.157:3300/0 conn(0x7f342815ab40 msgr2=0x7f34280ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:48.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 --2- 172.21.15.136:0/1030820718 >> v2:172.21.15.157:3300/0 conn(0x7f342815ab40 0x7f34280ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 -- 172.21.15.136:0/1030820718 >> v1:172.21.15.157:6789/0 conn(0x7f34280ca290 legacy=0x7f342815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.268+0000 7f342d3b4640 1 -- 172.21.15.136:0/1030820718 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342815dff0 con 0x7f34280b5060 2024-04-25T11:28:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f342d3b4640 1 --2- 172.21.15.136:0/1030820718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa27bc71852b165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f341ffff640 1 -- 172.21.15.136:0/1030820718 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240b5070 con 0x7f34280b5060 2024-04-25T11:28:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f341ffff640 1 -- 172.21.15.136:0/1030820718 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34240a8a50 con 0x7f34280b5060 2024-04-25T11:28:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f341ffff640 1 -- 172.21.15.136:0/1030820718 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240a8e20 con 0x7f34280b5060 2024-04-25T11:28:48.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f342f618640 1 -- 172.21.15.136:0/1030820718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 msgr2=0x7f34280c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.272+0000 7f342f618640 1 --2- 172.21.15.136:0/1030820718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401ed50 comp rx=0 tx=0).stop 2024-04-25T11:28:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 -- 172.21.15.136:0/1030820718 shutdown_connections 2024-04-25T11:28:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 --2- 172.21.15.136:0/1030820718 >> v2:172.21.15.157:3300/0 conn(0x7f342815ab40 0x7f34280ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 --2- 172.21.15.136:0/1030820718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f34280c4520 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 -- 172.21.15.136:0/1030820718 >> 172.21.15.136:0/1030820718 conn(0x7f3428070f70 msgr2=0x7f3428068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:48.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 -- 172.21.15.136:0/1030820718 shutdown_connections 2024-04-25T11:28:48.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 -- 172.21.15.136:0/1030820718 wait complete. 2024-04-25T11:28:48.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 Processor -- start 2024-04-25T11:28:48.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.276+0000 7f342f618640 1 -- start start 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f342814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 0x7f34281504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f342815d3f0 con 0x7f34280ca290 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3428159f10 con 0x7f342815ab40 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34280c4a60 con 0x7f34280b5060 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33246/0 (socket says 172.21.15.136:33246) 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 -- 172.21.15.136:0/3321642350 learned_addr learned my addr 172.21.15.136:0/3321642350 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342d3b4640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f342814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342dbb5640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 0x7f34281504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 -- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 msgr2=0x7f342814e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f342814e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 -- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 msgr2=0x7f34281504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 0x7f34281504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 -- 172.21.15.136:0/3321642350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342814e620 con 0x7f34280ca290 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342dbb5640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 0x7f34281504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342cbb3640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f341800c3a0 tx=0x7f341800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34180052d0 con 0x7f34280ca290 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3418005470 con 0x7f34280ca290 2024-04-25T11:28:48.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3418007260 con 0x7f34280ca290 2024-04-25T11:28:48.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342806e310 con 0x7f34280ca290 2024-04-25T11:28:48.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34281539b0 con 0x7f34280ca290 2024-04-25T11:28:48.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.280+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342806a350 con 0x7f34280ca290 2024-04-25T11:28:48.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.284+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3418007400 con 0x7f34280ca290 2024-04-25T11:28:48.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.284+0000 7f341dffb640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 0x7f340c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:48.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.284+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f341808efb0 con 0x7f34280ca290 2024-04-25T11:28:48.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.284+0000 7f342d3b4640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 0x7f340c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:48.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.284+0000 7f342d3b4640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 0x7f340c071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f34240b2000 tx=0x7f3424009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:48.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.288+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f341805b4b0 con 0x7f34280ca290 2024-04-25T11:28:48.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.512+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34280cadd0 con 0x7f34280ca290 2024-04-25T11:28:48.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.512+0000 7f341dffb640 1 -- 172.21.15.136:0/3321642350 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6094 (secure 0 0 0) 0x7f341805f7b0 con 0x7f34280ca290 2024-04-25T11:28:48.503 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:48.503 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:44.143342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238]},"id":1}]} 2024-04-25T11:28:48.503 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 34 2024-04-25T11:28:48.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 msgr2=0x7f340c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 0x7f340c071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f34240b2000 tx=0x7f3424009040 comp rx=0 tx=0).stop 2024-04-25T11:28:48.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 msgr2=0x7f342806ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:48.506 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f341800c3a0 tx=0x7f341800c860 comp rx=0 tx=0).stop 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 shutdown_connections 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f342815ab40 0x7f34281504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34280ca290 0x7f342806ddd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f340c06f400 0x7f340c071910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 --2- 172.21.15.136:0/3321642350 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34280b5060 0x7f342814e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 >> 172.21.15.136:0/3321642350 conn(0x7f3428070f70 msgr2=0x7f34280cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 shutdown_connections 2024-04-25T11:28:48.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:48.516+0000 7f342f618640 1 -- 172.21.15.136:0/3321642350 wait complete. 2024-04-25T11:28:48.528 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4269": { "addr": "172.21.15.157:6837/281980057", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 281980057, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 281980057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4269, "incarnation": 14, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 14 }, "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:28:44.143342+0000", "qdb_cluster": [ 4269, 4297, 4303, 5238 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 4269, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:28:50.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:50.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 Processor -- start 2024-04-25T11:28:50.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 -- start start 2024-04-25T11:28:50.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f54740c9b50 0x7f547415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740c8a20 con 0x7f547406d210 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740c8d00 con 0x7f54740ca010 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54796dd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f547406c670 con 0x7f54740c9b50 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54737fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f547406d210 0x7f547415c520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:58436/0 (socket says 172.21.15.136:58436) 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54737fe640 1 -- 172.21.15.136:0/2340532620 learned_addr learned my addr 172.21.15.136:0/2340532620 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f54727fc640 1 --2- 172.21.15.136:0/2340532620 >> v2:172.21.15.157:3300/0 conn(0x7f54740c9b50 0x7f547415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700175722 0 0) 0x7f54740c8a20 con 0x7f547406d210 2024-04-25T11:28:50.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.704+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f545c003530 con 0x7f547406d210 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 126547680 0 0) 0x7f545c003530 con 0x7f547406d210 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 >> v2:172.21.15.157:3300/0 conn(0x7f54740c9b50 msgr2=0x7f547415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 --2- 172.21.15.136:0/2340532620 >> v2:172.21.15.157:3300/0 conn(0x7f54740c9b50 0x7f547415bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 >> v1:172.21.15.157:6789/0 conn(0x7f54740ca010 legacy=0x7f547406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5474074d10 con 0x7f547406d210 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f546c001a20 con 0x7f547406d210 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 >> v1:172.21.15.136:6789/0 conn(0x7f547406d210 legacy=0x7f547415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c0039c0 0x7f545c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c004810 0x7f545c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f545c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f547406c670 con 0x7f545c0051d0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f545c00c170 con 0x7f545c0039c0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f545c00c370 con 0x7f545c004810 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f545c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5472ffd640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c0039c0 0x7f545c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54727fc640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c004810 0x7f545c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 -- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c004810 msgr2=0x7f545c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c004810 0x7f545c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 -- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c0039c0 msgr2=0x7f545c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c0039c0 0x7f545c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 -- 172.21.15.136:0/2340532620 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c003530 con 0x7f545c0051d0 2024-04-25T11:28:50.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54727fc640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c004810 0x7f545c004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 -- 172.21.15.136:0/2340532620 reap_dead start 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f54737fe640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f545c00b9d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f546c00a850 tx=0x7f546c01d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647576fa25d8018e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c00a4a0 con 0x7f545c0051d0 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546c0ad650 con 0x7f545c0051d0 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.708+0000 7f5471ffb640 1 -- 172.21.15.136:0/2340532620 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c0ada40 con 0x7f545c0051d0 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 msgr2=0x7f545c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f545c00b9d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f546c00a850 tx=0x7f546c01d2a0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- 172.21.15.136:0/2340532620 shutdown_connections 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 --2- 172.21.15.136:0/2340532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f545c00b9d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- 172.21.15.136:0/2340532620 >> 172.21.15.136:0/2340532620 conn(0x7f54740ca5a0 msgr2=0x7f54740ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:50.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- 172.21.15.136:0/2340532620 shutdown_connections 2024-04-25T11:28:50.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- 172.21.15.136:0/2340532620 wait complete. 2024-04-25T11:28:50.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 Processor -- start 2024-04-25T11:28:50.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.712+0000 7f54796dd640 1 -- start start 2024-04-25T11:28:50.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 0x7f5474152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5474153430 0x7f547414e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f547406cb00 con 0x7f545c0051d0 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f54740c9240 con 0x7f5474153430 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f547406cc80 con 0x7f54741e55a0 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33264/0 (socket says 172.21.15.136:33264) 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 -- 172.21.15.136:0/4052370469 learned_addr learned my addr 172.21.15.136:0/4052370469 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54727fc640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 0x7f5474152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54737fe640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5474153430 0x7f547414e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 -- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 msgr2=0x7f5474152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 0x7f5474152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 -- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5474153430 msgr2=0x7f547414e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5474153430 0x7f547414e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 -- 172.21.15.136:0/4052370469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741543f0 con 0x7f545c0051d0 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54727fc640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 0x7f5474152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f5472ffd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f546400e4d0 tx=0x7f546400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640052e0 con 0x7f545c0051d0 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5464018070 con 0x7f545c0051d0 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464003820 con 0x7f545c0051d0 2024-04-25T11:28:50.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54741f4be0 con 0x7f545c0051d0 2024-04-25T11:28:50.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5474154110 con 0x7f545c0051d0 2024-04-25T11:28:50.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.716+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547406a350 con 0x7f545c0051d0 2024-04-25T11:28:50.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.720+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f54640039c0 con 0x7f545c0051d0 2024-04-25T11:28:50.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.720+0000 7f545b7fe640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 0x7f5460071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:50.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.720+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f546408f3c0 con 0x7f545c0051d0 2024-04-25T11:28:50.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.720+0000 7f54727fc640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 0x7f5460071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:50.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.720+0000 7f54727fc640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 0x7f5460071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54741e5f60 tx=0x7f5468008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:50.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.728+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f546405bec0 con 0x7f545c0051d0 2024-04-25T11:28:50.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.952+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f547406acb0 con 0x7f545c0051d0 2024-04-25T11:28:50.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f545b7fe640 1 -- 172.21.15.136:0/4052370469 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6091 (secure 0 0 0) 0x7f5464005480 con 0x7f545c0051d0 2024-04-25T11:28:50.944 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:50.945 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:49.151930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:28:50.945 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 35 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 msgr2=0x7f5460071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 0x7f5460071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54741e5f60 tx=0x7f5468008040 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 msgr2=0x7f54741f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f546400e4d0 tx=0x7f546400e990 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 shutdown_connections 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5474153430 0x7f547414e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f546006f400 0x7f5460071910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f54741e55a0 0x7f5474152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 --2- 172.21.15.136:0/4052370469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f545c0051d0 0x7f54741f46a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 >> 172.21.15.136:0/4052370469 conn(0x7f54740ca5a0 msgr2=0x7f5474157240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 shutdown_connections 2024-04-25T11:28:50.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:50.956+0000 7f54796dd640 1 -- 172.21.15.136:0/4052370469 wait complete. 2024-04-25T11:28:50.967 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:51.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:28:51.755 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:28:52.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:53.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 Processor -- start 2024-04-25T11:28:53.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 -- start start 2024-04-25T11:28:53.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f43600c96e0 0x7f43601540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:53.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43600c9af0 con 0x7f43600c79e0 2024-04-25T11:28:53.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43600cb220 con 0x7f436015aae0 2024-04-25T11:28:53.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.136+0000 7f43673d0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f436006cc10 con 0x7f43600c96e0 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f43600c96e0 0x7f43601540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f43600c96e0 0x7f43601540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33276/0 (socket says 172.21.15.136:33276) 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 -- 172.21.15.136:0/1545373560 learned_addr learned my addr 172.21.15.136:0/1545373560 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 -- 172.21.15.136:0/1545373560 >> v1:172.21.15.157:6789/0 conn(0x7f436015aae0 legacy=0x7f4360164900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 -- 172.21.15.136:0/1545373560 >> v1:172.21.15.136:6789/0 conn(0x7f43600c79e0 legacy=0x7f436015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 -- 172.21.15.136:0/1545373560 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43600cb500 con 0x7f43600c96e0 2024-04-25T11:28:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f436516c640 1 --2- 172.21.15.136:0/1545373560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c96e0 0x7f43601540d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f436006cd90 tx=0x7f435c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d595ab7154133d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f4357fff640 1 -- 172.21.15.136:0/1545373560 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c0b5070 con 0x7f43600c96e0 2024-04-25T11:28:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f4357fff640 1 -- 172.21.15.136:0/1545373560 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435c0a89f0 con 0x7f43600c96e0 2024-04-25T11:28:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f4357fff640 1 -- 172.21.15.136:0/1545373560 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c0a8dc0 con 0x7f43600c96e0 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 -- 172.21.15.136:0/1545373560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c96e0 msgr2=0x7f43601540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 --2- 172.21.15.136:0/1545373560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c96e0 0x7f43601540d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f436006cd90 tx=0x7f435c01ecf0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 -- 172.21.15.136:0/1545373560 shutdown_connections 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 --2- 172.21.15.136:0/1545373560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c96e0 0x7f43601540d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 -- 172.21.15.136:0/1545373560 >> 172.21.15.136:0/1545373560 conn(0x7f436006f9d0 msgr2=0x7f436006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 -- 172.21.15.136:0/1545373560 shutdown_connections 2024-04-25T11:28:53.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.140+0000 7f43673d0640 1 -- 172.21.15.136:0/1545373560 wait complete. 2024-04-25T11:28:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 Processor -- start 2024-04-25T11:28:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 -- start start 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43600c96e0 0x7f4360147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f436015aae0 0x7f4360148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43600cf0f0 con 0x7f43600c79e0 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4360154610 con 0x7f43600c96e0 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f43673d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f43600ced40 con 0x7f436015aae0 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33290/0 (socket says 172.21.15.136:33290) 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 -- 172.21.15.136:0/2489861888 learned_addr learned my addr 172.21.15.136:0/2489861888 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 -- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f436015aae0 msgr2=0x7f4360148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f436015aae0 0x7f4360148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436496b640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43600c96e0 0x7f4360147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:53.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 -- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43600c96e0 msgr2=0x7f4360147cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43600c96e0 0x7f4360147cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 -- 172.21.15.136:0/2489861888 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43601f1190 con 0x7f43600c79e0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f436516c640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f435c0b3040 tx=0x7f435c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.144+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c0c0070 con 0x7f43600c79e0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f435c0b08e0 con 0x7f43600c79e0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43601f0770 con 0x7f43600c79e0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c0b5040 con 0x7f43600c79e0 2024-04-25T11:28:53.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4360157180 con 0x7f43600c79e0 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436006a350 con 0x7f43600c79e0 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f435c0c3070 con 0x7f43600c79e0 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f4355ffb640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 0x7f4344071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f435c12c700 con 0x7f43600c79e0 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f436496b640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 0x7f4344071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:53.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.148+0000 7f436496b640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 0x7f4344071910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f436006e5a0 tx=0x7f4350009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:53.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.152+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f435c018850 con 0x7f43600c79e0 2024-04-25T11:28:53.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.368+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43600c55b0 con 0x7f43600c79e0 2024-04-25T11:28:53.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.368+0000 7f4355ffb640 1 -- 172.21.15.136:0/2489861888 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6091 (secure 0 0 0) 0x7f435c0f90e0 con 0x7f43600c79e0 2024-04-25T11:28:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:53.359 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:49.151930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:28:53.359 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 35 2024-04-25T11:28:53.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 msgr2=0x7f4344071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 0x7f4344071910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f436006e5a0 tx=0x7f4350009040 comp rx=0 tx=0).stop 2024-04-25T11:28:53.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 msgr2=0x7f43601f0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:53.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f435c0b3040 tx=0x7f435c01ba90 comp rx=0 tx=0).stop 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 shutdown_connections 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f434406f400 0x7f4344071910 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f436015aae0 0x7f4360148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43600c96e0 0x7f4360147cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 --2- 172.21.15.136:0/2489861888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43600c79e0 0x7f43601f0c50 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 >> 172.21.15.136:0/2489861888 conn(0x7f436006f9d0 msgr2=0x7f4360153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 shutdown_connections 2024-04-25T11:28:53.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:53.372+0000 7f43673d0640 1 -- 172.21.15.136:0/2489861888 wait complete. 2024-04-25T11:28:53.382 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:28:55.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:55.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 Processor -- start 2024-04-25T11:28:55.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 -- start start 2024-04-25T11:28:55.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f44b414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f44b40ca720 0x7f44b40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44b40cbad0 con 0x7f44b40ce260 2024-04-25T11:28:55.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f44b40c8550 con 0x7f44b40ca720 2024-04-25T11:28:55.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44ba4c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44b40c7c20 con 0x7f44b40c8cd0 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f44b40ce260 0x7f44b40ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:34568/0 (socket says 172.21.15.136:34568) 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b37fe640 1 -- 172.21.15.136:0/646234355 learned_addr learned my addr 172.21.15.136:0/646234355 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b8a64640 1 --2- 172.21.15.136:0/646234355 >> v2:172.21.15.157:3300/0 conn(0x7f44b40ca720 0x7f44b40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b3fff640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f44b414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502825964 0 0) 0x7f44b40cbad0 con 0x7f44b40ce260 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f449c003530 con 0x7f44b40ce260 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1812105014 0 0) 0x7f449c003530 con 0x7f44b40ce260 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 msgr2=0x7f44b414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f44b414f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 >> v2:172.21.15.157:3300/0 conn(0x7f44b40ca720 msgr2=0x7f44b40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 --2- 172.21.15.136:0/646234355 >> v2:172.21.15.157:3300/0 conn(0x7f44b40ca720 0x7f44b40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.560+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b40c7f00 con 0x7f44b40ce260 2024-04-25T11:28:55.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f44a8001680 con 0x7f44b40ce260 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 >> v1:172.21.15.136:6789/0 conn(0x7f44b40ce260 legacy=0x7f44b40ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f449c0039c0 0x7f449c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f449c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c004d50 0x7f449c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f44b40c8550 con 0x7f449c0039c0 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44b40c7c20 con 0x7f449c004d50 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f449c00dc90 con 0x7f44b40c8cd0 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f449c0039c0 0x7f449c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b37fe640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f449c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b8a64640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c004d50 0x7f449c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 -- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 msgr2=0x7f449c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b40c8cd0 0x7f449c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 -- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c004d50 msgr2=0x7f449c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c004d50 0x7f449c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 -- 172.21.15.136:0/646234355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449c003530 con 0x7f449c0039c0 2024-04-25T11:28:55.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b8a64640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c004d50 0x7f449c005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b3fff640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f449c0039c0 0x7f449c003e50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009ae0 tx=0x7f44a401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44206233439dc838 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a40b2070 con 0x7f449c0039c0 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a40a5540 con 0x7f449c0039c0 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b2ffd640 1 -- 172.21.15.136:0/646234355 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a40096b0 con 0x7f449c0039c0 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44ba4c7640 1 -- 172.21.15.136:0/646234355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f449c0039c0 msgr2=0x7f449c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/646234355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f449c0039c0 0x7f449c003e50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009ae0 tx=0x7f44a401ca60 comp rx=0 tx=0).stop 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44b8a64640 1 -- 172.21.15.136:0/646234355 reap_dead start 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.564+0000 7f44ba4c7640 1 -- 172.21.15.136:0/646234355 shutdown_connections 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.568+0000 7f44ba4c7640 1 -- 172.21.15.136:0/646234355 >> 172.21.15.136:0/646234355 conn(0x7f44b406f9d0 msgr2=0x7f44b406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:55.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.568+0000 7f44ba4c7640 1 -- 172.21.15.136:0/646234355 shutdown_connections 2024-04-25T11:28:55.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.568+0000 7f44ba4c7640 1 -- 172.21.15.136:0/646234355 wait complete. 2024-04-25T11:28:55.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.568+0000 7f44ba4c7640 1 Processor -- start 2024-04-25T11:28:55.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.568+0000 7f44ba4c7640 1 -- start start 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c0039c0 0x7f44b41eea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b4148bb0 0x7f44b4148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f44b40cb720 con 0x7f44b4152380 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44b40c91e0 con 0x7f449c0039c0 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44b415ac80 con 0x7f44b4148bb0 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46620/0 (socket says 172.21.15.136:46620) 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 -- 172.21.15.136:0/795318130 learned_addr learned my addr 172.21.15.136:0/795318130 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:55.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b3fff640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c0039c0 0x7f44b41eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b8a64640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b4148bb0 0x7f44b4148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 -- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b4148bb0 msgr2=0x7f44b4148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b4148bb0 0x7f44b4148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 -- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c0039c0 msgr2=0x7f44b41eea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c0039c0 0x7f44b41eea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 -- 172.21.15.136:0/795318130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4149500 con 0x7f44b4152380 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b37fe640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f44a8011810 tx=0x7f44a8011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8010040 con 0x7f44b4152380 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a801a070 con 0x7f44b4152380 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b4151b70 con 0x7f44b4152380 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a800fba0 con 0x7f44b4152380 2024-04-25T11:28:55.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b40ccc90 con 0x7f44b4152380 2024-04-25T11:28:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b415a100 con 0x7f44b4152380 2024-04-25T11:28:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.572+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f44a801f460 con 0x7f44b4152380 2024-04-25T11:28:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.576+0000 7f44b0ff9640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 0x7f44a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.576+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f44a80927f0 con 0x7f44b4152380 2024-04-25T11:28:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.576+0000 7f44b3fff640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 0x7f44a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.576+0000 7f44b3fff640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 0x7f44a0071910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f44a40af000 tx=0x7f44a40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:55.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.580+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f44b4149500 con 0x7f44b4152380 2024-04-25T11:28:55.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.808+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44b406a350 con 0x7f44b4152380 2024-04-25T11:28:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44b0ff9640 1 -- 172.21.15.136:0/795318130 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6096 (secure 0 0 0) 0x7f44a805eb90 con 0x7f44b4152380 2024-04-25T11:28:55.801 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:55.801 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4269,4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:28:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 36 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 msgr2=0x7f44a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 0x7f44a0071910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f44a40af000 tx=0x7f44a40046d0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 msgr2=0x7f44b40cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f44a8011810 tx=0x7f44a8011cd0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 shutdown_connections 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f44a006f400 0x7f44a0071910 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44b4148bb0 0x7f44b4148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44b4152380 0x7f44b40cd0d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 --2- 172.21.15.136:0/795318130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449c0039c0 0x7f44b41eea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 >> 172.21.15.136:0/795318130 conn(0x7f44b406f9d0 msgr2=0x7f44b406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 shutdown_connections 2024-04-25T11:28:55.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:55.812+0000 7f44ba4c7640 1 -- 172.21.15.136:0/795318130 wait complete. 2024-04-25T11:28:55.825 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=3, replacing gid=4260 2024-04-25T11:28:55.825 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.e 2024-04-25T11:28:55.825 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-04-25T11:28:55.825 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-25T11:28:55.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-25T11:28:55.827 INFO:tasks.ceph.mds.e:Started 2024-04-25T11:28:55.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:28:55.928 INFO:tasks.ceph.mds.e.smithi136.stdout:starting mds.e at 2024-04-25T11:28:55.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 Processor -- start 2024-04-25T11:28:55.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 -- start start 2024-04-25T11:28:55.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ad80cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad815c6f0 con 0x7f5ad8160370 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ad815c990 con 0x7f5ad815ff60 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5adc911640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80c6310 con 0x7f5ad815c0d0 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5ad6d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5ad8160370 0x7f5ad8166a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:34594/0 (socket says 172.21.15.136:34594) 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/4013126843 learned_addr learned my addr 172.21.15.136:0/4013126843 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5ad659c640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ad80cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591826828 0 0) 0x7f5ad815c6f0 con 0x7f5ad8160370 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.000+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ac0003530 con 0x7f5ad8160370 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671845821 0 0) 0x7f5ad815c990 con 0x7f5ad815ff60 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad815c6f0 con 0x7f5ad815ff60 2024-04-25T11:28:55.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2912038168 0 0) 0x7f5ac0003530 con 0x7f5ad8160370 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 msgr2=0x7f5ad80cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ad80cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 >> v1:172.21.15.157:6789/0 conn(0x7f5ad815ff60 legacy=0x7f5ad80ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad815cc30 con 0x7f5ad8160370 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f5ad0001a00 con 0x7f5ad8160370 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 >> v1:172.21.15.136:6789/0 conn(0x7f5ad8160370 legacy=0x7f5ad8166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ac0003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac0004130 0x7f5ac0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004d50 0x7f5ac00051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80c6310 con 0x7f5ac0004130 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac000bb90 con 0x7f5ac0004d50 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac000bd90 con 0x7f5ad815c0d0 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac0004130 0x7f5ac0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad6d9d640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004d50 0x7f5ac00051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad659c640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ac0003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:55.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 -- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 msgr2=0x7f5ac0003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad815c0d0 0x7f5ac0003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 -- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004d50 msgr2=0x7f5ac00051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:28:55.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004d50 0x7f5ac00051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:55.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 -- 172.21.15.136:0/4013126843 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac0003530 con 0x7f5ac0004130 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad5d9b640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac0004130 0x7f5ac0004810 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80c6640 tx=0x7f5acc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4ba550cd78fb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc001520 con 0x7f5ac0004130 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5acc0052d0 con 0x7f5ac0004130 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.004+0000 7f5ad559a640 1 -- 172.21.15.136:0/4013126843 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc003840 con 0x7f5ac0004130 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- 172.21.15.136:0/4013126843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac0004130 msgr2=0x7f5ac0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 --2- 172.21.15.136:0/4013126843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac0004130 0x7f5ac0004810 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80c6640 tx=0x7f5acc01c520 comp rx=0 tx=0).stop 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/4013126843 reap_dead start 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- 172.21.15.136:0/4013126843 shutdown_connections 2024-04-25T11:28:55.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- 172.21.15.136:0/4013126843 >> 172.21.15.136:0/4013126843 conn(0x7f5ad806f9d0 msgr2=0x7f5ad806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:55.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- 172.21.15.136:0/4013126843 shutdown_connections 2024-04-25T11:28:55.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- 172.21.15.136:0/4013126843 wait complete. 2024-04-25T11:28:55.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 Processor -- start 2024-04-25T11:28:55.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.008+0000 7f5adc911640 1 -- start start 2024-04-25T11:28:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004130 0x7f5ad81f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad814ec00 0x7f5ad814bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad815cf10 con 0x7f5ad814f150 2024-04-25T11:28:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad815fa40 con 0x7f5ac0004130 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80c9170 con 0x7f5ad814ec00 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46648/0 (socket says 172.21.15.136:46648) 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/2415632141 learned_addr learned my addr 172.21.15.136:0/2415632141 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad659c640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004130 0x7f5ad81f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad5d9b640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad814ec00 0x7f5ad814bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad814ec00 msgr2=0x7f5ad814bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad814ec00 0x7f5ad814bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004130 msgr2=0x7f5ad81f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004130 0x7f5ad81f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 -- 172.21.15.136:0/2415632141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad814ea10 con 0x7f5ad814f150 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ad6d9d640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0001f30 tx=0x7f5ad00076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:56.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad001c070 con 0x7f5ad814f150 2024-04-25T11:28:56.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad0006070 con 0x7f5ad814f150 2024-04-25T11:28:56.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad8156a20 con 0x7f5ad814f150 2024-04-25T11:28:56.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad0010040 con 0x7f5ad814f150 2024-04-25T11:28:56.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad81f2db0 con 0x7f5ad814f150 2024-04-25T11:28:56.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad806a350 con 0x7f5ad814f150 2024-04-25T11:28:56.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ad001f070 con 0x7f5ad814f150 2024-04-25T11:28:56.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.012+0000 7f5ac6ffd640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 0x7f5ab8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:28:56.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.016+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f5ad00920f0 con 0x7f5ad814f150 2024-04-25T11:28:56.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.016+0000 7f5ad659c640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 0x7f5ab8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:28:56.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.016+0000 7f5ad659c640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 0x7f5ab8071940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009b40 tx=0x7f5ac8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:28:56.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.020+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ad8156a20 con 0x7f5ad814f150 2024-04-25T11:28:56.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.240+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad8075d90 con 0x7f5ad814f150 2024-04-25T11:28:56.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5ac6ffd640 1 -- 172.21.15.136:0/2415632141 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f5ad005e510 con 0x7f5ad814f150 2024-04-25T11:28:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:28:56.233 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:28:56.233 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:28:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 msgr2=0x7f5ab8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 0x7f5ab8071940 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5ac8009b40 tx=0x7f5ac8009190 comp rx=0 tx=0).stop 2024-04-25T11:28:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 msgr2=0x7f5ad814f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:28:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0001f30 tx=0x7f5ad00076b0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 shutdown_connections 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad814f150 0x7f5ad814f6a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5ab806f430 0x7f5ab8071940 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ad814ec00 0x7f5ad814bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 --2- 172.21.15.136:0/2415632141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac0004130 0x7f5ad81f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 >> 172.21.15.136:0/2415632141 conn(0x7f5ad806f9d0 msgr2=0x7f5ad81541e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 shutdown_connections 2024-04-25T11:28:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:28:56.244+0000 7f5adc911640 1 -- 172.21.15.136:0/2415632141 wait complete. 2024-04-25T11:28:56.255 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-04-25T11:28:56.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2024-04-25T11:28:58.502 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:28:58.511+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:29:21.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:29:21.855 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:29:51.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:29:51.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 Processor -- start 2024-04-25T11:29:51.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 -- start start 2024-04-25T11:29:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9f640c8cd0 0x7f9f6414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f640ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f640cbad0 con 0x7f9f640ce260 2024-04-25T11:29:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9f640c8550 con 0x7f9f640c8cd0 2024-04-25T11:29:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f6a28d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f640c7c20 con 0x7f9f640ca720 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f63fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9f640c8cd0 0x7f9f6414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f5bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9f640ce260 0x7f9f640ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50746/0 (socket says 172.21.15.136:50746) 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.447+0000 7f9f5bfff640 1 -- 172.21.15.136:0/1831641117 learned_addr learned my addr 172.21.15.136:0/1831641117 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6882a640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f640ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f63fff640 1 -- 172.21.15.136:0/1831641117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 msgr2=0x7f9f640ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f63fff640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f640ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f63fff640 1 -- 172.21.15.136:0/1831641117 >> v1:172.21.15.136:6789/0 conn(0x7f9f640ce260 legacy=0x7f9f640ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f637fe640 1 -- 172.21.15.136:0/1831641117 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428239471 0 0) 0x7f9f640cbad0 con 0x7f9f640ce260 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f63fff640 1 -- 172.21.15.136:0/1831641117 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f640c7f00 con 0x7f9f640c8cd0 2024-04-25T11:29:51.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6882a640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f640ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:29:51.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f63fff640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f6414f680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9f640c86d0 tx=0x7f9f5401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586fcb36c579d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f637fe640 1 -- 172.21.15.136:0/1831641117 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540b5070 con 0x7f9f640c8cd0 2024-04-25T11:29:51.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f637fe640 1 -- 172.21.15.136:0/1831641117 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f540a8990 con 0x7f9f640c8cd0 2024-04-25T11:29:51.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f637fe640 1 -- 172.21.15.136:0/1831641117 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540a8d60 con 0x7f9f640c8cd0 2024-04-25T11:29:51.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 -- 172.21.15.136:0/1831641117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 msgr2=0x7f9f6414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f6414f680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9f640c86d0 tx=0x7f9f5401ec90 comp rx=0 tx=0).stop 2024-04-25T11:29:51.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 -- 172.21.15.136:0/1831641117 shutdown_connections 2024-04-25T11:29:51.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f640ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/1831641117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f6414f680 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 -- 172.21.15.136:0/1831641117 >> 172.21.15.136:0/1831641117 conn(0x7f9f6406f9d0 msgr2=0x7f9f6406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:51.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 -- 172.21.15.136:0/1831641117 shutdown_connections 2024-04-25T11:29:51.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.451+0000 7f9f6a28d640 1 -- 172.21.15.136:0/1831641117 wait complete. 2024-04-25T11:29:51.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 Processor -- start 2024-04-25T11:29:51.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 -- start start 2024-04-25T11:29:51.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f6414ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f640ce260 0x7f9f6414b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640cb720 con 0x7f9f640c8cd0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6414fbc0 con 0x7f9f640ce260 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f6a28d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f64155b50 con 0x7f9f640ca720 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f63fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f63fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44682/0 (socket says 172.21.15.136:44682) 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f63fff640 1 -- 172.21.15.136:0/192499555 learned_addr learned my addr 172.21.15.136:0/192499555 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.455+0000 7f9f5bfff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f6414ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f6882a640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f640ce260 0x7f9f6414b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 -- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 msgr2=0x7f9f6414ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f6414ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 -- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f640ce260 msgr2=0x7f9f6414b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f640ce260 0x7f9f6414b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 -- 172.21.15.136:0/192499555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f64155680 con 0x7f9f640c8cd0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f63fff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9f5401d340 tx=0x7f9f5401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540be070 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f540b0880 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f540b5040 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6415b1a0 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f64154940 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6406a350 con 0x7f9f640c8cd0 2024-04-25T11:29:51.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9f540c1070 con 0x7f9f640c8cd0 2024-04-25T11:29:51.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 0x7f9f44071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f9f5412bf40 con 0x7f9f640c8cd0 2024-04-25T11:29:51.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.459+0000 7f9f5bfff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 0x7f9f44071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.463+0000 7f9f5bfff640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 0x7f9f44071940 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9f641531b0 tx=0x7f9f4c007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.467+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f540187f0 con 0x7f9f640c8cd0 2024-04-25T11:29:51.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.719+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f64156790 con 0x7f9f640c8cd0 2024-04-25T11:29:51.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.719+0000 7f9f617fa640 1 -- 172.21.15.136:0/192499555 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f9f540f8a40 con 0x7f9f640c8cd0 2024-04-25T11:29:51.711 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:29:51.711 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:29:51.711 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:29:51.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 msgr2=0x7f9f44071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 0x7f9f44071940 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9f641531b0 tx=0x7f9f4c007c10 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 msgr2=0x7f9f64155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9f5401d340 tx=0x7f9f5401ba30 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 shutdown_connections 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f640ce260 0x7f9f6414b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4406f430 0x7f9f44071940 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f640ca720 0x7f9f6414ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 --2- 172.21.15.136:0/192499555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f640c8cd0 0x7f9f64155140 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 >> 172.21.15.136:0/192499555 conn(0x7f9f6406f9d0 msgr2=0x7f9f6406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 shutdown_connections 2024-04-25T11:29:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.723+0000 7f9f6a28d640 1 -- 172.21.15.136:0/192499555 wait complete. 2024-04-25T11:29:51.737 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2024-04-25T11:29:51.738 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T11:29:51.738 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:29:51.758 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T11:29:51.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:29:51.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 Processor -- start 2024-04-25T11:29:51.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 -- start start 2024-04-25T11:29:51.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 0x7f39e014f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f39e00ce260 0x7f39e00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f39e00ca010 0x7f39e00ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f39e00c8550 con 0x7f39e00ca010 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f39e00cbad0 con 0x7f39e00ce260 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39e5dfa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f39e00c7c20 con 0x7f39e00c8cd0 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.919+0000 7f39dffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f39e00ca010 0x7f39e00ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f39e00ca010 0x7f39e00ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44694/0 (socket says 172.21.15.136:44694) 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 -- 172.21.15.136:0/3841047190 learned_addr learned my addr 172.21.15.136:0/3841047190 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39df7fe640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 0x7f39e014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39deffd640 1 --2- 172.21.15.136:0/3841047190 >> v2:172.21.15.157:3300/0 conn(0x7f39e00ce260 0x7f39e00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 -- 172.21.15.136:0/3841047190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 msgr2=0x7f39e014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 0x7f39e014f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 -- 172.21.15.136:0/3841047190 >> v2:172.21.15.157:3300/0 conn(0x7f39e00ce260 msgr2=0x7f39e00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 --2- 172.21.15.136:0/3841047190 >> v2:172.21.15.157:3300/0 conn(0x7f39e00ce260 0x7f39e00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 -- 172.21.15.136:0/3841047190 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e00c7f00 con 0x7f39e00ca010 2024-04-25T11:29:51.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39df7fe640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 0x7f39e014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:29:51.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39dffff640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e00ca420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f39e00706f0 tx=0x7f39d801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08acaec8df976bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39de7fc640 1 -- 172.21.15.136:0/3841047190 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d80b4070 con 0x7f39e00ca010 2024-04-25T11:29:51.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39de7fc640 1 -- 172.21.15.136:0/3841047190 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d80af400 con 0x7f39e00ca010 2024-04-25T11:29:51.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.923+0000 7f39de7fc640 1 -- 172.21.15.136:0/3841047190 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d80af7d0 con 0x7f39e00ca010 2024-04-25T11:29:51.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- 172.21.15.136:0/3841047190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 msgr2=0x7f39e00ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e00ca420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f39e00706f0 tx=0x7f39d801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- 172.21.15.136:0/3841047190 shutdown_connections 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e00ca420 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/3841047190 >> v2:172.21.15.157:3300/0 conn(0x7f39e00ce260 0x7f39e00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/3841047190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00c8cd0 0x7f39e014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- 172.21.15.136:0/3841047190 >> 172.21.15.136:0/3841047190 conn(0x7f39e006f9d0 msgr2=0x7f39e006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- 172.21.15.136:0/3841047190 shutdown_connections 2024-04-25T11:29:51.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- 172.21.15.136:0/3841047190 wait complete. 2024-04-25T11:29:51.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 Processor -- start 2024-04-25T11:29:51.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- start start 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39e00c8cd0 0x7f39e0150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00ce260 0x7f39e0151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f39e00cb720 con 0x7f39e00ca010 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f39e00ca960 con 0x7f39e00c8cd0 2024-04-25T11:29:51.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.927+0000 7f39e5dfa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f39e01f2da0 con 0x7f39e00ce260 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44702/0 (socket says 172.21.15.136:44702) 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 -- 172.21.15.136:0/594938355 learned_addr learned my addr 172.21.15.136:0/594938355 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39df7fe640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39e00c8cd0 0x7f39e0150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39dffff640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00ce260 0x7f39e0151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 -- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00ce260 msgr2=0x7f39e0151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00ce260 0x7f39e0151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 -- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39e00c8cd0 msgr2=0x7f39e0150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39e00c8cd0 0x7f39e0150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 -- 172.21.15.136:0/594938355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e014f7b0 con 0x7f39e00ca010 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39deffd640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f39d400d620 tx=0x7f39d400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d400b620 con 0x7f39e00ca010 2024-04-25T11:29:51.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d4019070 con 0x7f39e00ca010 2024-04-25T11:29:51.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e014fa90 con 0x7f39e00ca010 2024-04-25T11:29:51.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d400b9e0 con 0x7f39e00ca010 2024-04-25T11:29:51.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e01f2ae0 con 0x7f39e00ca010 2024-04-25T11:29:51.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e014d0d0 con 0x7f39e00ca010 2024-04-25T11:29:51.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f39d400bb80 con 0x7f39e00ca010 2024-04-25T11:29:51.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 0x7f39cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:51.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f39d408fb50 con 0x7f39e00ca010 2024-04-25T11:29:51.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.931+0000 7f39df7fe640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 0x7f39cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:51.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.935+0000 7f39df7fe640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 0x7f39cc071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39d000b990 tx=0x7f39d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:51.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:51.939+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f39d405c650 con 0x7f39e00ca010 2024-04-25T11:29:51.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:29:51.982 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:29:52.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.187+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39e006a350 con 0x7f39e00ca010 2024-04-25T11:29:52.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39c7fff640 1 -- 172.21.15.136:0/594938355 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f39d405bf50 con 0x7f39e00ca010 2024-04-25T11:29:52.180 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:29:52.180 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:29:52.180 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:29:52.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 msgr2=0x7f39cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:52.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 0x7f39cc071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39d000b990 tx=0x7f39d0008040 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 msgr2=0x7f39e0158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f39d400d620 tx=0x7f39d400dae0 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 shutdown_connections 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f39cc06f400 0x7f39cc071910 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f39e00ce260 0x7f39e0151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39e00ca010 0x7f39e0158a90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 --2- 172.21.15.136:0/594938355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f39e00c8cd0 0x7f39e0150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 >> 172.21.15.136:0/594938355 conn(0x7f39e006f9d0 msgr2=0x7f39e006cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 shutdown_connections 2024-04-25T11:29:52.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:52.191+0000 7f39e5dfa640 1 -- 172.21.15.136:0/594938355 wait complete. 2024-04-25T11:29:52.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:29:54.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:29:54.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.367+0000 7f0d6a563640 1 Processor -- start 2024-04-25T11:29:54.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.367+0000 7f0d6a563640 1 -- start start 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d6a563640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d64074980 0x7f0d640c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d6a563640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0d640c4170 0x7f0d641636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d6a563640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d640c3750 con 0x7f0d6414f840 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d6a563640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6406dc00 con 0x7f0d64074980 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d6a563640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6414fc50 con 0x7f0d640c4170 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d64074980 0x7f0d640c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d64074980 0x7f0d640c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44710/0 (socket says 172.21.15.136:44710) 2024-04-25T11:29:54.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 -- 172.21.15.136:0/3995023035 learned_addr learned my addr 172.21.15.136:0/3995023035 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:54.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d68b00640 1 --2- 172.21.15.136:0/3995023035 >> v2:172.21.15.157:3300/0 conn(0x7f0d640c4170 0x7f0d641636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 -- 172.21.15.136:0/3995023035 >> v2:172.21.15.157:3300/0 conn(0x7f0d640c4170 msgr2=0x7f0d641636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:54.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 --2- 172.21.15.136:0/3995023035 >> v2:172.21.15.157:3300/0 conn(0x7f0d640c4170 0x7f0d641636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 -- 172.21.15.136:0/3995023035 >> v1:172.21.15.157:6789/0 conn(0x7f0d6414f840 legacy=0x7f0d64073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:54.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.371+0000 7f0d637fe640 1 -- 172.21.15.136:0/3995023035 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d640c3190 con 0x7f0d64074980 2024-04-25T11:29:54.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d637fe640 1 --2- 172.21.15.136:0/3995023035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64074980 0x7f0d640c3c30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800bb10 tx=0x7f0d5801ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f9cafc5b1445f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:54.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d62ffd640 1 -- 172.21.15.136:0/3995023035 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d580b4070 con 0x7f0d64074980 2024-04-25T11:29:54.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d62ffd640 1 -- 172.21.15.136:0/3995023035 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d580a7a70 con 0x7f0d64074980 2024-04-25T11:29:54.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d62ffd640 1 -- 172.21.15.136:0/3995023035 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d580a7e40 con 0x7f0d64074980 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- 172.21.15.136:0/3995023035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64074980 msgr2=0x7f0d640c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 --2- 172.21.15.136:0/3995023035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64074980 0x7f0d640c3c30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800bb10 tx=0x7f0d5801ebb0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- 172.21.15.136:0/3995023035 shutdown_connections 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 --2- 172.21.15.136:0/3995023035 >> v2:172.21.15.157:3300/0 conn(0x7f0d640c4170 0x7f0d641636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 --2- 172.21.15.136:0/3995023035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d64074980 0x7f0d640c3c30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- 172.21.15.136:0/3995023035 >> 172.21.15.136:0/3995023035 conn(0x7f0d640ca5a0 msgr2=0x7f0d640ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- 172.21.15.136:0/3995023035 shutdown_connections 2024-04-25T11:29:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- 172.21.15.136:0/3995023035 wait complete. 2024-04-25T11:29:54.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 Processor -- start 2024-04-25T11:29:54.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.375+0000 7f0d6a563640 1 -- start start 2024-04-25T11:29:54.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 0x7f0d641f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6414f840 0x7f0d640c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d64074d90 con 0x7f0d640c4170 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0d64159c10 con 0x7f0d6414f840 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d64153380 con 0x7f0d64074980 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44712/0 (socket says 172.21.15.136:44712) 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 -- 172.21.15.136:0/581836095 learned_addr learned my addr 172.21.15.136:0/581836095 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d68b00640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6414f840 0x7f0d640c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d63fff640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 0x7f0d641f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 -- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 msgr2=0x7f0d641f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 0x7f0d641f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 -- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6414f840 msgr2=0x7f0d640c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6414f840 0x7f0d640c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 -- 172.21.15.136:0/581836095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d641f54e0 con 0x7f0d640c4170 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d63fff640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 0x7f0d641f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:29:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d637fe640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b2c0 tx=0x7f0d5801b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:54.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d580b4040 con 0x7f0d640c4170 2024-04-25T11:29:54.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d580c1070 con 0x7f0d640c4170 2024-04-25T11:29:54.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d640c63f0 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d580b9460 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d641f47a0 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.379+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d640c6160 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.383+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0d580b7070 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.383+0000 7f0d60ff9640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 0x7f0d50071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.383+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f0d5812c6a0 con 0x7f0d640c4170 2024-04-25T11:29:54.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.383+0000 7f0d63fff640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 0x7f0d50071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:54.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.383+0000 7f0d63fff640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 0x7f0d50071910 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400a850 tx=0x7f0d54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:54.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.387+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d580187a0 con 0x7f0d640c4170 2024-04-25T11:29:54.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.595+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d6406a350 con 0x7f0d640c4170 2024-04-25T11:29:54.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.595+0000 7f0d60ff9640 1 -- 172.21.15.136:0/581836095 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f0d580f9110 con 0x7f0d640c4170 2024-04-25T11:29:54.586 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:29:54.586 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:29:54.587 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 msgr2=0x7f0d50071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 0x7f0d50071910 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400a850 tx=0x7f0d54008040 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 msgr2=0x7f0d64147a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b2c0 tx=0x7f0d5801b950 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 shutdown_connections 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6414f840 0x7f0d640c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d640c4170 0x7f0d64147a80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d5006f400 0x7f0d50071910 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 --2- 172.21.15.136:0/581836095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d64074980 0x7f0d641f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 >> 172.21.15.136:0/581836095 conn(0x7f0d640ca5a0 msgr2=0x7f0d6414e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 shutdown_connections 2024-04-25T11:29:54.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:54.599+0000 7f0d6a563640 1 -- 172.21.15.136:0/581836095 wait complete. 2024-04-25T11:29:54.609 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:29:56.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:29:56.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 Processor -- start 2024-04-25T11:29:56.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 -- start start 2024-04-25T11:29:56.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6f08154060 0x7f6f080c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815a5f0 0x7f6f080c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f0815aa00 0x7f6f080c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f6f080c8fd0 con 0x7f6f0815aa00 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6f080c8d40 con 0x7f6f08154060 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0f763640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f080c3880 con 0x7f6f0815a5f0 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f0815aa00 0x7f6f080c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f0815aa00 0x7f6f080c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60640/0 (socket says 172.21.15.136:60640) 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 -- 172.21.15.136:0/1476196183 learned_addr learned my addr 172.21.15.136:0/1476196183 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0d4ff640 1 --2- 172.21.15.136:0/1476196183 >> v2:172.21.15.157:3300/0 conn(0x7f6f08154060 0x7f6f080c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0ccfe640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815a5f0 0x7f6f080c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 -- 172.21.15.136:0/1476196183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815a5f0 msgr2=0x7f6f080c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815a5f0 0x7f6f080c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 -- 172.21.15.136:0/1476196183 >> v2:172.21.15.157:3300/0 conn(0x7f6f08154060 msgr2=0x7f6f080c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 --2- 172.21.15.136:0/1476196183 >> v2:172.21.15.157:3300/0 conn(0x7f6f08154060 0x7f6f080c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.791+0000 7f6f0dd00640 1 -- 172.21.15.136:0/1476196183 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0806d9b0 con 0x7f6f0815aa00 2024-04-25T11:29:56.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0dd00640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f0815aa00 0x7f6f080c76f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6f0806db80 tx=0x7f6f0001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1594202c09482881 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:56.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6efffff640 1 -- 172.21.15.136:0/1476196183 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000b5070 con 0x7f6f0815aa00 2024-04-25T11:29:56.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6efffff640 1 -- 172.21.15.136:0/1476196183 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f000a89d0 con 0x7f6f0815aa00 2024-04-25T11:29:56.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6efffff640 1 -- 172.21.15.136:0/1476196183 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000a8da0 con 0x7f6f0815aa00 2024-04-25T11:29:56.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 -- 172.21.15.136:0/1476196183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f0815aa00 msgr2=0x7f6f080c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:56.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f0815aa00 0x7f6f080c76f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6f0806db80 tx=0x7f6f0001ed10 comp rx=0 tx=0).stop 2024-04-25T11:29:56.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 -- 172.21.15.136:0/1476196183 shutdown_connections 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f0815aa00 0x7f6f080c76f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 --2- 172.21.15.136:0/1476196183 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815a5f0 0x7f6f080c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 --2- 172.21.15.136:0/1476196183 >> v2:172.21.15.157:3300/0 conn(0x7f6f08154060 0x7f6f080c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 -- 172.21.15.136:0/1476196183 >> 172.21.15.136:0/1476196183 conn(0x7f6f080ca5a0 msgr2=0x7f6f080ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 -- 172.21.15.136:0/1476196183 shutdown_connections 2024-04-25T11:29:56.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.795+0000 7f6f0f763640 1 -- 172.21.15.136:0/1476196183 wait complete. 2024-04-25T11:29:56.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 Processor -- start 2024-04-25T11:29:56.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 -- start start 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f0815a5f0 0x7f6f081486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815aa00 0x7f6f0814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0806df30 con 0x7f6f08154060 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08154470 con 0x7f6f0815a5f0 2024-04-25T11:29:56.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0f763640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0815a0d0 con 0x7f6f0815aa00 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60654/0 (socket says 172.21.15.136:60654) 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 -- 172.21.15.136:0/358919015 learned_addr learned my addr 172.21.15.136:0/358919015 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0ccfe640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f0815a5f0 0x7f6f081486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0dd00640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815aa00 0x7f6f0814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 -- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815aa00 msgr2=0x7f6f0814b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815aa00 0x7f6f0814b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 -- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f0815a5f0 msgr2=0x7f6f081486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f0815a5f0 0x7f6f081486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.799+0000 7f6f0d4ff640 1 -- 172.21.15.136:0/358919015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f081f1410 con 0x7f6f08154060 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6f0d4ff640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6f04007e60 tx=0x7f6f0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04014620 con 0x7f6f08154060 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f04019070 con 0x7f6f08154060 2024-04-25T11:29:56.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f081f06f0 con 0x7f6f08154060 2024-04-25T11:29:56.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040149e0 con 0x7f6f08154060 2024-04-25T11:29:56.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081560e0 con 0x7f6f08154060 2024-04-25T11:29:56.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0806a350 con 0x7f6f08154060 2024-04-25T11:29:56.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6f04014b80 con 0x7f6f08154060 2024-04-25T11:29:56.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 0x7f6eec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:56.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f6f0408ff10 con 0x7f6f08154060 2024-04-25T11:29:56.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.803+0000 7f6f0ccfe640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 0x7f6eec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:56.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.807+0000 7f6f0ccfe640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 0x7f6eec071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f08154850 tx=0x7f6ef8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:56.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:56.811+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6f0405ca10 con 0x7f6f08154060 2024-04-25T11:29:57.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.071+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f080c85e0 con 0x7f6f08154060 2024-04-25T11:29:57.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.071+0000 7f6efdffb640 1 -- 172.21.15.136:0/358919015 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f6f0405c310 con 0x7f6f08154060 2024-04-25T11:29:57.061 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:29:57.061 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:29:57.061 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 msgr2=0x7f6eec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 0x7f6eec071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f08154850 tx=0x7f6ef8008040 comp rx=0 tx=0).stop 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 msgr2=0x7f6f081f0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6f04007e60 tx=0x7f6f0400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 shutdown_connections 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6eec06f400 0x7f6eec071910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:57.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6f0815aa00 0x7f6f0814b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:57.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6f0815a5f0 0x7f6f081486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:57.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 --2- 172.21.15.136:0/358919015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f08154060 0x7f6f081f0ed0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:57.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 >> 172.21.15.136:0/358919015 conn(0x7f6f080ca5a0 msgr2=0x7f6f08158860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:57.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 shutdown_connections 2024-04-25T11:29:57.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:57.075+0000 7f6f0f763640 1 -- 172.21.15.136:0/358919015 wait complete. 2024-04-25T11:29:57.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:29:59.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:29:59.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.275+0000 7ffadc705640 1 Processor -- start 2024-04-25T11:29:59.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.275+0000 7ffadc705640 1 -- start start 2024-04-25T11:29:59.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffadc705640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ffad40ca290 0x7ffad40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:59.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffadc705640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad415d750 con 0x7ffad40b5060 2024-04-25T11:29:59.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffadc705640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad415da30 con 0x7ffad415a430 2024-04-25T11:29:59.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffadc705640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ffad415dd10 con 0x7ffad40ca290 2024-04-25T11:29:59.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad9ca0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ffad40ca290 0x7ffad40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffada4a1640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffad40b5060 0x7ffad40c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47184/0 (socket says 172.21.15.136:47184) 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffada4a1640 1 -- 172.21.15.136:0/2284990689 learned_addr learned my addr 172.21.15.136:0/2284990689 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad9ca0640 1 -- 172.21.15.136:0/2284990689 >> v1:172.21.15.157:6789/0 conn(0x7ffad415a430 legacy=0x7ffad415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad9ca0640 1 -- 172.21.15.136:0/2284990689 >> v1:172.21.15.136:6789/0 conn(0x7ffad40b5060 legacy=0x7ffad40c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad9ca0640 1 -- 172.21.15.136:0/2284990689 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad415dff0 con 0x7ffad40ca290 2024-04-25T11:29:59.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad949f640 1 -- 172.21.15.136:0/2284990689 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589412122 0 0) 0x7ffad415d750 con 0x7ffad40b5060 2024-04-25T11:29:59.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.279+0000 7ffad9ca0640 1 --2- 172.21.15.136:0/2284990689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad40ca6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097c0 tx=0x7ffac401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551e3f1495071755 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:59.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffad949f640 1 -- 172.21.15.136:0/2284990689 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40b5070 con 0x7ffad40ca290 2024-04-25T11:29:59.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffad949f640 1 -- 172.21.15.136:0/2284990689 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac40a8950 con 0x7ffad40ca290 2024-04-25T11:29:59.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffad949f640 1 -- 172.21.15.136:0/2284990689 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40a8d20 con 0x7ffad40ca290 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- 172.21.15.136:0/2284990689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 msgr2=0x7ffad40ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 --2- 172.21.15.136:0/2284990689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad40ca6a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097c0 tx=0x7ffac401ec50 comp rx=0 tx=0).stop 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- 172.21.15.136:0/2284990689 shutdown_connections 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 --2- 172.21.15.136:0/2284990689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad40ca6a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- 172.21.15.136:0/2284990689 >> 172.21.15.136:0/2284990689 conn(0x7ffad4070f70 msgr2=0x7ffad4068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- 172.21.15.136:0/2284990689 shutdown_connections 2024-04-25T11:29:59.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- 172.21.15.136:0/2284990689 wait complete. 2024-04-25T11:29:59.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 Processor -- start 2024-04-25T11:29:59.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.283+0000 7ffadc705640 1 -- start start 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffad40b5060 0x7ffad4147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffad415a430 0x7ffad415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffad415d3f0 con 0x7ffad40ca290 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffad4159f10 con 0x7ffad415a430 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffad4165790 con 0x7ffad40b5060 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60680/0 (socket says 172.21.15.136:60680) 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 -- 172.21.15.136:0/3093270098 learned_addr learned my addr 172.21.15.136:0/3093270098 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffada4a1640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffad40b5060 0x7ffad4147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadaca2640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffad415a430 0x7ffad415c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 -- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffad40b5060 msgr2=0x7ffad4147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffad40b5060 0x7ffad4147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 -- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffad415a430 msgr2=0x7ffad415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffad415a430 0x7ffad415c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 -- 172.21.15.136:0/3093270098 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad4148080 con 0x7ffad40ca290 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffad9ca0640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffac40b3040 tx=0x7ffac401d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40c1070 con 0x7ffad40ca290 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac40b6040 con 0x7ffad40ca290 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 -- 172.21.15.136:0/3093270098 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad414c8e0 con 0x7ffad40ca290 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac40b5050 con 0x7ffad40ca290 2024-04-25T11:29:59.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffadc705640 1 -- 172.21.15.136:0/3093270098 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad4148290 con 0x7ffad40ca290 2024-04-25T11:29:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab00052a0 con 0x7ffad40ca290 2024-04-25T11:29:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.287+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffac40b0790 con 0x7ffad40ca290 2024-04-25T11:29:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.291+0000 7ffacaffd640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 0x7ffab8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:29:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.291+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7ffac412cb20 con 0x7ffad40ca290 2024-04-25T11:29:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.291+0000 7ffada4a1640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 0x7ffab8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:29:59.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.291+0000 7ffada4a1640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 0x7ffab8071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffad0000f70 tx=0x7ffad0007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:29:59.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.295+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffac40187b0 con 0x7ffad40ca290 2024-04-25T11:29:59.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.527+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffab0005860 con 0x7ffad40ca290 2024-04-25T11:29:59.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.527+0000 7ffacaffd640 1 -- 172.21.15.136:0/3093270098 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7ffac40f8f50 con 0x7ffad40ca290 2024-04-25T11:29:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:29:59.517 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:29:59.518 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:29:59.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 msgr2=0x7ffab8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 0x7ffab8071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffad0000f70 tx=0x7ffad0007c10 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 msgr2=0x7ffad415be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffac40b3040 tx=0x7ffac401d490 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 shutdown_connections 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffad415a430 0x7ffad415c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad40ca290 0x7ffad415be90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffab806f430 0x7ffab8071940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 --2- 172.21.15.136:0/3093270098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffad40b5060 0x7ffad4147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 >> 172.21.15.136:0/3093270098 conn(0x7ffad4070f70 msgr2=0x7ffad40cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 shutdown_connections 2024-04-25T11:29:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:29:59.531+0000 7ffac8ff9640 1 -- 172.21.15.136:0/3093270098 wait complete. 2024-04-25T11:29:59.541 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:30:01.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:01.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.727+0000 7f353d669640 1 Processor -- start 2024-04-25T11:30:01.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.727+0000 7f353d669640 1 -- start start 2024-04-25T11:30:01.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f353d669640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 0x7f353806d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f353d669640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f353815c480 0x7f353806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f353d669640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380b4480 con 0x7f35380b5810 2024-04-25T11:30:01.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f353d669640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f35380b4a90 con 0x7f353815c480 2024-04-25T11:30:01.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f353d669640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f353815c990 con 0x7f353815c070 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f353815c480 0x7f353806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f353815c480 0x7f353806da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60684/0 (socket says 172.21.15.136:60684) 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 -- 172.21.15.136:0/2668775926 learned_addr learned my addr 172.21.15.136:0/2668775926 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f352e7fc640 1 --2- 172.21.15.136:0/2668775926 >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 0x7f353806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 -- 172.21.15.136:0/2668775926 >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 msgr2=0x7f353806d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 --2- 172.21.15.136:0/2668775926 >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 0x7f353806d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 -- 172.21.15.136:0/2668775926 >> v1:172.21.15.157:6789/0 conn(0x7f35380b5810 legacy=0x7f35380b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 -- 172.21.15.136:0/2668775926 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353815cc70 con 0x7f353815c480 2024-04-25T11:30:01.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f352e7fc640 1 --2- 172.21.15.136:0/2668775926 >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 0x7f353806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:01.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35377fe640 1 --2- 172.21.15.136:0/2668775926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c480 0x7f353806da50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f353815c8c0 tx=0x7f353001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2dd5d705ae383f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:01.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.731+0000 7f35367fc640 1 -- 172.21.15.136:0/2668775926 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35300b4070 con 0x7f353815c480 2024-04-25T11:30:01.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f35367fc640 1 -- 172.21.15.136:0/2668775926 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35300af400 con 0x7f353815c480 2024-04-25T11:30:01.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f35367fc640 1 -- 172.21.15.136:0/2668775926 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35300af7d0 con 0x7f353815c480 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- 172.21.15.136:0/2668775926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c480 msgr2=0x7f353806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 --2- 172.21.15.136:0/2668775926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c480 0x7f353806da50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f353815c8c0 tx=0x7f353001f180 comp rx=0 tx=0).stop 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- 172.21.15.136:0/2668775926 shutdown_connections 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 --2- 172.21.15.136:0/2668775926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c480 0x7f353806da50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 --2- 172.21.15.136:0/2668775926 >> v2:172.21.15.157:3300/0 conn(0x7f353815c070 0x7f353806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- 172.21.15.136:0/2668775926 >> 172.21.15.136:0/2668775926 conn(0x7f35380743a0 msgr2=0x7f35380d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- 172.21.15.136:0/2668775926 shutdown_connections 2024-04-25T11:30:01.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- 172.21.15.136:0/2668775926 wait complete. 2024-04-25T11:30:01.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 Processor -- start 2024-04-25T11:30:01.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.735+0000 7f353d669640 1 -- start start 2024-04-25T11:30:01.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 0x7f3538156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 0x7f35380ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f353806e030 con 0x7f353815c070 2024-04-25T11:30:01.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f353815cf50 con 0x7f35380b5810 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35380cbee0 con 0x7f353815c480 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60698/0 (socket says 172.21.15.136:60698) 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 -- 172.21.15.136:0/305674753 learned_addr learned my addr 172.21.15.136:0/305674753 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f3536ffd640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 0x7f3538156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f35377fe640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 0x7f35380ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 -- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 msgr2=0x7f35380ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 0x7f35380ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 -- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 msgr2=0x7f3538156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 0x7f3538156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 -- 172.21.15.136:0/305674753 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3538156a80 con 0x7f353815c070 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f35377fe640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 0x7f35380ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:01.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f3536ffd640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 0x7f3538156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352e7fc640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f352000c7f0 tx=0x7f352000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520015070 con 0x7f353815c070 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3538157470 con 0x7f353815c070 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3520007820 con 0x7f353815c070 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520007c10 con 0x7f353815c070 2024-04-25T11:30:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381f3050 con 0x7f353815c070 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.739+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353806a350 con 0x7f353815c070 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.743+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3520018070 con 0x7f353815c070 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.743+0000 7f352ffff640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 0x7f3518071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.743+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f352008f700 con 0x7f353815c070 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.743+0000 7f3536ffd640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 0x7f3518071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:01.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.743+0000 7f3536ffd640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 0x7f3518071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35280098b0 tx=0x7f3528008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:01.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.747+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f352005c0f0 con 0x7f353815c070 2024-04-25T11:30:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.963+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35380cd5f0 con 0x7f353815c070 2024-04-25T11:30:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.963+0000 7f352ffff640 1 -- 172.21.15.136:0/305674753 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f352005b9f0 con 0x7f353815c070 2024-04-25T11:30:01.954 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:01.954 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:30:01.954 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 msgr2=0x7f3518071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 0x7f3518071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35280098b0 tx=0x7f3528008040 comp rx=0 tx=0).stop 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 msgr2=0x7f35380c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f352000c7f0 tx=0x7f352000ccb0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 shutdown_connections 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f351806f400 0x7f3518071910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f353815c480 0x7f35380ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f353815c070 0x7f35380c6d30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 --2- 172.21.15.136:0/305674753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f35380b5810 0x7f3538156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 >> 172.21.15.136:0/305674753 conn(0x7f35380743a0 msgr2=0x7f35380cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 shutdown_connections 2024-04-25T11:30:01.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:01.967+0000 7f353d669640 1 -- 172.21.15.136:0/305674753 wait complete. 2024-04-25T11:30:01.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:30:03.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:04.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 Processor -- start 2024-04-25T11:30:04.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 -- start start 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9694156330 0x7f969415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96940b5ba0 0x7f96940ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f969415d010 con 0x7f96940cdee0 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f96941602f0 con 0x7f96940b5ba0 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9699bf5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f96940ccb80 con 0x7f9694156330 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96940b5ba0 0x7f96940ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96940b5ba0 0x7f96940ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60702/0 (socket says 172.21.15.136:60702) 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 -- 172.21.15.136:0/3215040582 learned_addr learned my addr 172.21.15.136:0/3215040582 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f96937fe640 1 --2- 172.21.15.136:0/3215040582 >> v2:172.21.15.157:3300/0 conn(0x7f9694156330 0x7f969415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 -- 172.21.15.136:0/3215040582 >> v2:172.21.15.157:3300/0 conn(0x7f9694156330 msgr2=0x7f969415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 --2- 172.21.15.136:0/3215040582 >> v2:172.21.15.157:3300/0 conn(0x7f9694156330 0x7f969415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 -- 172.21.15.136:0/3215040582 >> v1:172.21.15.157:6789/0 conn(0x7f96940cdee0 legacy=0x7f96940b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.155+0000 7f9693fff640 1 -- 172.21.15.136:0/3215040582 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9694156740 con 0x7f96940b5ba0 2024-04-25T11:30:04.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9693fff640 1 --2- 172.21.15.136:0/3215040582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940b5ba0 0x7f96940ce2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f96940cf010 tx=0x7f968c01efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7305fa7b4f167a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:04.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f96927fc640 1 -- 172.21.15.136:0/3215040582 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c0a8bd0 con 0x7f96940b5ba0 2024-04-25T11:30:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f96927fc640 1 -- 172.21.15.136:0/3215040582 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f968c0a8d70 con 0x7f96940b5ba0 2024-04-25T11:30:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f96927fc640 1 -- 172.21.15.136:0/3215040582 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c0b0610 con 0x7f96940b5ba0 2024-04-25T11:30:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 -- 172.21.15.136:0/3215040582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940b5ba0 msgr2=0x7f96940ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 --2- 172.21.15.136:0/3215040582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940b5ba0 0x7f96940ce2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f96940cf010 tx=0x7f968c01efc0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 -- 172.21.15.136:0/3215040582 shutdown_connections 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 --2- 172.21.15.136:0/3215040582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940b5ba0 0x7f96940ce2f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 --2- 172.21.15.136:0/3215040582 >> v2:172.21.15.157:3300/0 conn(0x7f9694156330 0x7f969415cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 -- 172.21.15.136:0/3215040582 >> 172.21.15.136:0/3215040582 conn(0x7f969406d6c0 msgr2=0x7f969406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 -- 172.21.15.136:0/3215040582 shutdown_connections 2024-04-25T11:30:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.159+0000 7f9699bf5640 1 -- 172.21.15.136:0/3215040582 wait complete. 2024-04-25T11:30:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 Processor -- start 2024-04-25T11:30:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 -- start start 2024-04-25T11:30:04.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96940b5ba0 0x7f9694154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9694156330 0x7f969415ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f969415fca0 con 0x7f96940cdee0 2024-04-25T11:30:04.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f969415f9c0 con 0x7f9694156330 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9699bf5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f969415ff80 con 0x7f96940b5ba0 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60708/0 (socket says 172.21.15.136:60708) 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 -- 172.21.15.136:0/2836839867 learned_addr learned my addr 172.21.15.136:0/2836839867 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9693fff640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9694156330 0x7f969415ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f96937fe640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96940b5ba0 0x7f9694154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 -- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96940b5ba0 msgr2=0x7f9694154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96940b5ba0 0x7f9694154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 -- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9694156330 msgr2=0x7f969415ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9694156330 0x7f969415ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 -- 172.21.15.136:0/2836839867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96941e5920 con 0x7f96940cdee0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.163+0000 7f9692ffd640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f968800b410 tx=0x7f968800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688015030 con 0x7f96940cdee0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9688004020 con 0x7f96940cdee0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96941f6a60 con 0x7f96940cdee0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688004410 con 0x7f96940cdee0 2024-04-25T11:30:04.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96941e5bc0 con 0x7f96940cdee0 2024-04-25T11:30:04.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9694155410 con 0x7f96940cdee0 2024-04-25T11:30:04.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9688018070 con 0x7f96940cdee0 2024-04-25T11:30:04.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 0x7f9680071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:04.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7f968808f350 con 0x7f96940cdee0 2024-04-25T11:30:04.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f96937fe640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 0x7f9680071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:04.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.167+0000 7f96937fe640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 0x7f9680071910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f968400b910 tx=0x7f9684008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:04.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.175+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f968805bd30 con 0x7f96940cdee0 2024-04-25T11:30:04.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.459+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f969406a350 con 0x7f96940cdee0 2024-04-25T11:30:04.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.459+0000 7f967bfff640 1 -- 172.21.15.136:0/2836839867 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7f968805b630 con 0x7f96940cdee0 2024-04-25T11:30:04.451 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:04.451 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:30:04.451 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:30:04.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 msgr2=0x7f9680071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:04.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 0x7f9680071910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f968400b910 tx=0x7f9684008040 comp rx=0 tx=0).stop 2024-04-25T11:30:04.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 msgr2=0x7f96941f6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:04.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f968800b410 tx=0x7f968800b8d0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 shutdown_connections 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9694156330 0x7f969415ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96940cdee0 0x7f96941f6520 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f968006f400 0x7f9680071910 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 --2- 172.21.15.136:0/2836839867 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96940b5ba0 0x7f9694154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 >> 172.21.15.136:0/2836839867 conn(0x7f969406d6c0 msgr2=0x7f96940c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 shutdown_connections 2024-04-25T11:30:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:04.463+0000 7f9699bf5640 1 -- 172.21.15.136:0/2836839867 wait complete. 2024-04-25T11:30:04.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:30:06.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:06.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 Processor -- start 2024-04-25T11:30:06.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 -- start start 2024-04-25T11:30:06.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc15cea0 0x7ff7fc0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7fc0cbce0 0x7ff7fc0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc073510 con 0x7ff7fc156390 2024-04-25T11:30:06.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff7fc15c300 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff80203b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc0ca0e0 con 0x7ff7fc15cea0 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7fc0cbce0 0x7ff7fc0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fb7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff7fc156390 0x7ff7fc15c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57642/0 (socket says 172.21.15.136:57642) 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fb7fe640 1 -- 172.21.15.136:0/1428327653 learned_addr learned my addr 172.21.15.136:0/1428327653 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7f3fff640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc15cea0 0x7ff7fc0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fbfff640 1 -- 172.21.15.136:0/1428327653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc15cea0 msgr2=0x7ff7fc0cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fbfff640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc15cea0 0x7ff7fc0cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fbfff640 1 -- 172.21.15.136:0/1428327653 >> v1:172.21.15.136:6789/0 conn(0x7ff7fc156390 legacy=0x7ff7fc15c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7faffd640 1 -- 172.21.15.136:0/1428327653 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 66884674 0 0) 0x7ff7fc073510 con 0x7ff7fc156390 2024-04-25T11:30:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.663+0000 7ff7fbfff640 1 -- 172.21.15.136:0/1428327653 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0cb1e0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff7fbfff640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0cc0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc15c480 tx=0x7ff7f401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edccf2f11d9352b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:06.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff7faffd640 1 -- 172.21.15.136:0/1428327653 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40b2070 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff7faffd640 1 -- 172.21.15.136:0/1428327653 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f40a58e0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff7faffd640 1 -- 172.21.15.136:0/1428327653 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40a5cb0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 -- 172.21.15.136:0/1428327653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 msgr2=0x7ff7fc0cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0cc0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc15c480 tx=0x7ff7f401c950 comp rx=0 tx=0).stop 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 -- 172.21.15.136:0/1428327653 shutdown_connections 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0cc0f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 --2- 172.21.15.136:0/1428327653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc15cea0 0x7ff7fc0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 -- 172.21.15.136:0/1428327653 >> 172.21.15.136:0/1428327653 conn(0x7ff7fc068c40 msgr2=0x7ff7fc072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 -- 172.21.15.136:0/1428327653 shutdown_connections 2024-04-25T11:30:06.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.667+0000 7ff80203b640 1 -- 172.21.15.136:0/1428327653 wait complete. 2024-04-25T11:30:06.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 Processor -- start 2024-04-25T11:30:06.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- start start 2024-04-25T11:30:06.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc156390 0x7ff7fc15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1f5d50 0x7ff7fc1467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0cc7b0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0cc630 con 0x7ff7fc1f5d50 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc1567a0 con 0x7ff7fc156390 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52068/0 (socket says 172.21.15.136:52068) 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 -- 172.21.15.136:0/3179446355 learned_addr learned my addr 172.21.15.136:0/3179446355 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 -- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc156390 msgr2=0x7ff7fc15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7f3fff640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc156390 0x7ff7fc15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fbfff640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1f5d50 0x7ff7fc1467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc156390 0x7ff7fc15ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 -- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1f5d50 msgr2=0x7ff7fc1467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1f5d50 0x7ff7fc1467e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 -- 172.21.15.136:0/3179446355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0ca5b0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7fb7fe640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c480 tx=0x7ff7ec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec018070 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7ec007500 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc147640 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec004a30 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.671+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc147c00 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7fc06a350 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff7ec01b070 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff7f8ff9640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 0x7ff7dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6993+0+0 (secure 0 0 0) 0x7ff7ec08ea70 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff7f3fff640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 0x7ff7dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:06.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.675+0000 7ff7f3fff640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 0x7ff7dc071910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0caab0 tx=0x7ff7e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:06.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.679+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff7ec05b570 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.903+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7fc0caae0 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.903+0000 7ff7f8ff9640 1 -- 172.21.15.136:0/3179446355 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6923 (secure 0 0 0) 0x7ff7ec05ae70 con 0x7ff7fc0cbce0 2024-04-25T11:30:06.894 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:06.894 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5517,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:28:54.194097+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":4269,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"b","rank":2,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.157:6837/281980057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":281980057},{"type":"v1","addr":"172.21.15.157:6837","nonce":281980057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297,4269]},"id":1}]} 2024-04-25T11:30:06.894 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 37 2024-04-25T11:30:06.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 msgr2=0x7ff7dc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 0x7ff7dc071910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0caab0 tx=0x7ff7e4008040 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 msgr2=0x7ff7fc0c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00c480 tx=0x7ff7ec00c940 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 shutdown_connections 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7fc1f5d50 0x7ff7fc1467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7dc06f400 0x7ff7dc071910 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7fc156390 0x7ff7fc15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 --2- 172.21.15.136:0/3179446355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7fc0cbce0 0x7ff7fc0c4da0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 >> 172.21.15.136:0/3179446355 conn(0x7ff7fc068c40 msgr2=0x7ff7fc15c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 shutdown_connections 2024-04-25T11:30:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:06.907+0000 7ff80203b640 1 -- 172.21.15.136:0/3179446355 wait complete. 2024-04-25T11:30:06.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:30:08.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:09.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 Processor -- start 2024-04-25T11:30:09.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 -- start start 2024-04-25T11:30:09.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f73780b5810 0x7f73780b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737806d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f737815c480 0x7f737806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f73780b4a90 con 0x7f73780b5810 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f73780b4480 con 0x7f737815c480 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737f48b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f737815c990 con 0x7f737815c070 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f73780b5810 0x7f73780b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f73780b5810 0x7f73780b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52082/0 (socket says 172.21.15.136:52082) 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 -- 172.21.15.136:0/1955337880 learned_addr learned my addr 172.21.15.136:0/1955337880 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737ca26640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737da28640 1 --2- 172.21.15.136:0/1955337880 >> v2:172.21.15.157:3300/0 conn(0x7f737815c480 0x7f737806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 -- 172.21.15.136:0/1955337880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 msgr2=0x7f737806d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737806d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.079+0000 7f737d227640 1 -- 172.21.15.136:0/1955337880 >> v2:172.21.15.157:3300/0 conn(0x7f737815c480 msgr2=0x7f737806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737d227640 1 --2- 172.21.15.136:0/1955337880 >> v2:172.21.15.157:3300/0 conn(0x7f737815c480 0x7f737806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737d227640 1 -- 172.21.15.136:0/1955337880 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737815cc70 con 0x7f73780b5810 2024-04-25T11:30:09.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737d227640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73780b4f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f73780b7540 tx=0x7f737401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798722ee47f1e958 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f736ffff640 1 -- 172.21.15.136:0/1955337880 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740b5070 con 0x7f73780b5810 2024-04-25T11:30:09.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f736ffff640 1 -- 172.21.15.136:0/1955337880 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73740a89d0 con 0x7f73780b5810 2024-04-25T11:30:09.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f736ffff640 1 -- 172.21.15.136:0/1955337880 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740a8da0 con 0x7f73780b5810 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 -- 172.21.15.136:0/1955337880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 msgr2=0x7f73780b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73780b4f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f73780b7540 tx=0x7f737401ed10 comp rx=0 tx=0).stop 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 -- 172.21.15.136:0/1955337880 shutdown_connections 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 --2- 172.21.15.136:0/1955337880 >> v2:172.21.15.157:3300/0 conn(0x7f737815c480 0x7f737806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 --2- 172.21.15.136:0/1955337880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73780b4f20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 -- 172.21.15.136:0/1955337880 >> 172.21.15.136:0/1955337880 conn(0x7f73780743a0 msgr2=0x7f73780d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:09.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 -- 172.21.15.136:0/1955337880 shutdown_connections 2024-04-25T11:30:09.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.083+0000 7f737f48b640 1 -- 172.21.15.136:0/1955337880 wait complete. 2024-04-25T11:30:09.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 Processor -- start 2024-04-25T11:30:09.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 -- start start 2024-04-25T11:30:09.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f737815c480 0x7f73781e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f737806e030 con 0x7f73780b5810 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f737815cf50 con 0x7f737815c480 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737f48b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73780b5460 con 0x7f737815c070 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737d227640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737d227640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52098/0 (socket says 172.21.15.136:52098) 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.087+0000 7f737d227640 1 -- 172.21.15.136:0/513417618 learned_addr learned my addr 172.21.15.136:0/513417618 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737da28640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f737815c480 0x7f73781e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737ca26640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 -- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 msgr2=0x7f737815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737815ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 -- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f737815c480 msgr2=0x7f73781e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f737815c480 0x7f73781e6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 -- 172.21.15.136:0/513417618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737815f4b0 con 0x7f73780b5810 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737d227640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f737401d3c0 tx=0x7f737401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740b5070 con 0x7f73780b5810 2024-04-25T11:30:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73740b0a50 con 0x7f73780b5810 2024-04-25T11:30:09.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740b0e20 con 0x7f73780b5810 2024-04-25T11:30:09.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73781599f0 con 0x7f73780b5810 2024-04-25T11:30:09.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73781e80a0 con 0x7f73780b5810 2024-04-25T11:30:09.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737806a350 con 0x7f73780b5810 2024-04-25T11:30:09.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f73740c2060 con 0x7f73780b5810 2024-04-25T11:30:09.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 0x7f735c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f737412c9c0 con 0x7f73780b5810 2024-04-25T11:30:09.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.091+0000 7f737ca26640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 0x7f735c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.095+0000 7f737ca26640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 0x7f735c071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73781547e0 tx=0x7f7368008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.099+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f737412ee20 con 0x7f73780b5810 2024-04-25T11:30:09.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.363+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73780cd5f0 con 0x7f73780b5810 2024-04-25T11:30:09.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.363+0000 7f736dffb640 1 -- 172.21.15.136:0/513417618 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6093 (secure 0 0 0) 0x7f7374018870 con 0x7f73780b5810 2024-04-25T11:30:09.352 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:09.352 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:08.878183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:30:09.353 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 38 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 msgr2=0x7f735c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 0x7f735c071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73781547e0 tx=0x7f7368008040 comp rx=0 tx=0).stop 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 msgr2=0x7f73781e7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f737401d3c0 tx=0x7f737401bab0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 shutdown_connections 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f737815c480 0x7f73781e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f735c06f400 0x7f735c071910 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f737815c070 0x7f737815ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 --2- 172.21.15.136:0/513417618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73780b5810 0x7f73781e7770 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 >> 172.21.15.136:0/513417618 conn(0x7f73780743a0 msgr2=0x7f7378155c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 shutdown_connections 2024-04-25T11:30:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.367+0000 7f737f48b640 1 -- 172.21.15.136:0/513417618 wait complete. 2024-04-25T11:30:09.376 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-25T11:30:09.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:30:09.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:09.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.535+0000 7f05ae21c640 1 Processor -- start 2024-04-25T11:30:09.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.535+0000 7f05ae21c640 1 -- start start 2024-04-25T11:30:09.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05ae21c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f05a814f840 0x7f05a8074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05ae21c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 0x7f05a8073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05ae21c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a80c3750 con 0x7f05a80c3c30 2024-04-25T11:30:09.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05ae21c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f05a806dc00 con 0x7f05a814f840 2024-04-25T11:30:09.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05ae21c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f05a814fc50 con 0x7f05a8073200 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f05a814f840 0x7f05a8074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f05a814f840 0x7f05a8074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52100/0 (socket says 172.21.15.136:52100) 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 -- 172.21.15.136:0/957422946 learned_addr learned my addr 172.21.15.136:0/957422946 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a7fff640 1 --1- 172.21.15.136:0/957422946 >> v1:172.21.15.136:6789/0 conn(0x7f05a80c3c30 0x7f05a80c4040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57650/0 (socket says 172.21.15.136:57650) 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f059ffff640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 0x7f05a8073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 -- 172.21.15.136:0/957422946 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 msgr2=0x7f05a8073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 0x7f05a8073610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 -- 172.21.15.136:0/957422946 >> v1:172.21.15.136:6789/0 conn(0x7f05a80c3c30 legacy=0x7f05a80c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 -- 172.21.15.136:0/957422946 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a80c3190 con 0x7f05a814f840 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a6ffd640 1 -- 172.21.15.136:0/957422946 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410665418 0 0) 0x7f05a80c3750 con 0x7f05a80c3c30 2024-04-25T11:30:09.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f059ffff640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 0x7f05a8073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:09.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.539+0000 7f05a77fe640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a814f840 0x7f05a8074980 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0598009870 tx=0x7f059801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756c11d9be6177b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05a6ffd640 1 -- 172.21.15.136:0/957422946 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980b2070 con 0x7f05a814f840 2024-04-25T11:30:09.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05a6ffd640 1 -- 172.21.15.136:0/957422946 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05980a9070 con 0x7f05a814f840 2024-04-25T11:30:09.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05a6ffd640 1 -- 172.21.15.136:0/957422946 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980ad880 con 0x7f05a814f840 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- 172.21.15.136:0/957422946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a814f840 msgr2=0x7f05a8074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a814f840 0x7f05a8074980 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0598009870 tx=0x7f059801c560 comp rx=0 tx=0).stop 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- 172.21.15.136:0/957422946 shutdown_connections 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a8073200 0x7f05a8073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 --2- 172.21.15.136:0/957422946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a814f840 0x7f05a8074980 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- 172.21.15.136:0/957422946 >> 172.21.15.136:0/957422946 conn(0x7f05a80ca5a0 msgr2=0x7f05a80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- 172.21.15.136:0/957422946 shutdown_connections 2024-04-25T11:30:09.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- 172.21.15.136:0/957422946 wait complete. 2024-04-25T11:30:09.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 Processor -- start 2024-04-25T11:30:09.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.543+0000 7f05ae21c640 1 -- start start 2024-04-25T11:30:09.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 0x7f05a813f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a814f840 0x7f05a813e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f05a81597f0 con 0x7f05a8073200 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8153380 con 0x7f05a814f840 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f05a8159970 con 0x7f05a80c3c30 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52114/0 (socket says 172.21.15.136:52114) 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 -- 172.21.15.136:0/1109602979 learned_addr learned my addr 172.21.15.136:0/1109602979 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f059ffff640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 0x7f05a813f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a7fff640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a814f840 0x7f05a813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 -- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 msgr2=0x7f05a813f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 0x7f05a813f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 -- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a814f840 msgr2=0x7f05a813e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a814f840 0x7f05a813e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 -- 172.21.15.136:0/1109602979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a81470e0 con 0x7f05a8073200 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f059ffff640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 0x7f05a813f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a77fe640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f05980b0040 tx=0x7f0598019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980bb070 con 0x7f05a8073200 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05980addd0 con 0x7f05a8073200 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a80c6580 con 0x7f05a8073200 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05980b2040 con 0x7f05a8073200 2024-04-25T11:30:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a80c6b00 con 0x7f05a8073200 2024-04-25T11:30:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a806a350 con 0x7f05a8073200 2024-04-25T11:30:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.547+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f05980be070 con 0x7f05a8073200 2024-04-25T11:30:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.551+0000 7f05a4ff9640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 0x7f0588071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.551+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f0598128780 con 0x7f05a8073200 2024-04-25T11:30:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.551+0000 7f059ffff640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 0x7f0588071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:09.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.551+0000 7f059ffff640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 0x7f0588071940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f05a815d3b0 tx=0x7f0590008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:09.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.555+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0598016150 con 0x7f05a8073200 2024-04-25T11:30:09.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.795+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a80c2ed0 con 0x7f05a8073200 2024-04-25T11:30:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.795+0000 7f05a4ff9640 1 -- 172.21.15.136:0/1109602979 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6093 (secure 0 0 0) 0x7f05980f5230 con 0x7f05a8073200 2024-04-25T11:30:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:09.785 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:08.878183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:30:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 38 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 msgr2=0x7f0588071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 0x7f0588071940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f05a815d3b0 tx=0x7f0590008040 comp rx=0 tx=0).stop 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 msgr2=0x7f05a8146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f05980b0040 tx=0x7f0598019300 comp rx=0 tx=0).stop 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 shutdown_connections 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a814f840 0x7f05a813e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f430 0x7f0588071940 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a80c3c30 0x7f05a813f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 --2- 172.21.15.136:0/1109602979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a8073200 0x7f05a8146ba0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:09.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 >> 172.21.15.136:0/1109602979 conn(0x7f05a80ca5a0 msgr2=0x7f05a8144850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:09.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 shutdown_connections 2024-04-25T11:30:09.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:09.799+0000 7f05ae21c640 1 -- 172.21.15.136:0/1109602979 wait complete. 2024-04-25T11:30:09.816 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:30:08.878183+0000", "qdb_cluster": [ 4297, 4303, 5238, 5385 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:30:11.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:11.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 Processor -- start 2024-04-25T11:30:11.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 -- start start 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5bb0154060 0x7f5bb00c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb00c8d40 con 0x7f5bb015aa00 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb00c3880 con 0x7f5bb015a5f0 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5bb59dc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb00c8fd0 con 0x7f5bb0154060 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baeffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5bb0154060 0x7f5bb00c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5bb015aa00 0x7f5bb0164700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57656/0 (socket says 172.21.15.136:57656) 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baf7fe640 1 -- 172.21.15.136:0/1323076471 learned_addr learned my addr 172.21.15.136:0/1323076471 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baeffd640 1 -- 172.21.15.136:0/1323076471 >> v1:172.21.15.157:6789/0 conn(0x7f5bb015a5f0 legacy=0x7f5bb00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baeffd640 1 -- 172.21.15.136:0/1323076471 >> v1:172.21.15.136:6789/0 conn(0x7f5bb015aa00 legacy=0x7f5bb0164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:11.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:11.999+0000 7f5baeffd640 1 -- 172.21.15.136:0/1323076471 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb006d9b0 con 0x7f5bb0154060 2024-04-25T11:30:11.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5baeffd640 1 --2- 172.21.15.136:0/1323076471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0154060 0x7f5bb00c8800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00097c0 tx=0x7f5ba001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e417cd2f9ce45a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:11.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bae7fc640 1 -- 172.21.15.136:0/1323076471 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00b5070 con 0x7f5bb0154060 2024-04-25T11:30:11.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bae7fc640 1 -- 172.21.15.136:0/1323076471 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba00a8950 con 0x7f5bb0154060 2024-04-25T11:30:11.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bae7fc640 1 -- 172.21.15.136:0/1323076471 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00a8d20 con 0x7f5bb0154060 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1323076471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0154060 msgr2=0x7f5bb00c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1323076471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0154060 0x7f5bb00c8800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00097c0 tx=0x7f5ba001ec50 comp rx=0 tx=0).stop 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1323076471 shutdown_connections 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1323076471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb0154060 0x7f5bb00c8800 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1323076471 >> 172.21.15.136:0/1323076471 conn(0x7f5bb00ca5a0 msgr2=0x7f5bb00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1323076471 shutdown_connections 2024-04-25T11:30:11.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.003+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1323076471 wait complete. 2024-04-25T11:30:11.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 Processor -- start 2024-04-25T11:30:11.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- start start 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 0x7f5bb014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bb015aa00 0x7f5bb01e5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb006df30 con 0x7f5bb015a5f0 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0154470 con 0x7f5bb015aa00 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb015a0d0 con 0x7f5bb0154060 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52128/0 (socket says 172.21.15.136:52128) 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 -- 172.21.15.136:0/1866990676 learned_addr learned my addr 172.21.15.136:0/1866990676 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5baf7fe640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bb015aa00 0x7f5bb01e5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5baeffd640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 0x7f5bb014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 -- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 msgr2=0x7f5bb014e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:11.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 0x7f5bb014e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 -- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bb015aa00 msgr2=0x7f5bb01e5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bb015aa00 0x7f5bb01e5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 -- 172.21.15.136:0/1866990676 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01f6000 con 0x7f5bb015a5f0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5baeffd640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 0x7f5bb014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5ba7fff640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800b3c0 tx=0x7f5b9800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98011830 con 0x7f5bb015a5f0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b98016070 con 0x7f5bb015a5f0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb01f62e0 con 0x7f5bb015a5f0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98011c30 con 0x7f5bb015a5f0 2024-04-25T11:30:12.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.007+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01e3f50 con 0x7f5bb015a5f0 2024-04-25T11:30:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb006a350 con 0x7f5bb015a5f0 2024-04-25T11:30:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5b9801b460 con 0x7f5bb015a5f0 2024-04-25T11:30:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5bacff9640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 0x7f5b90071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f5b9808f480 con 0x7f5bb015a5f0 2024-04-25T11:30:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5baeffd640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 0x7f5b90071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:12.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.011+0000 7f5baeffd640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 0x7f5b90071910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00b2000 tx=0x7f5ba0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:12.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.015+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5b9805bea0 con 0x7f5bb015a5f0 2024-04-25T11:30:12.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.251+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bb00c85e0 con 0x7f5bb015a5f0 2024-04-25T11:30:12.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.251+0000 7f5bacff9640 1 -- 172.21.15.136:0/1866990676 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6093 (secure 0 0 0) 0x7f5b9805b7a0 con 0x7f5bb015a5f0 2024-04-25T11:30:12.240 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:12.240 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:08.878183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:30:12.240 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 38 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 msgr2=0x7f5b90071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 0x7f5b90071910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00b2000 tx=0x7f5ba0008040 comp rx=0 tx=0).stop 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 msgr2=0x7f5bb00ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800b3c0 tx=0x7f5b9800b880 comp rx=0 tx=0).stop 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 shutdown_connections 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bb015aa00 0x7f5bb01e5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bb015a5f0 0x7f5bb00ccac0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5b9006f400 0x7f5b90071910 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 --2- 172.21.15.136:0/1866990676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5bb0154060 0x7f5bb014e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:12.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 >> 172.21.15.136:0/1866990676 conn(0x7f5bb00ca5a0 msgr2=0x7f5bb01520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:12.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 shutdown_connections 2024-04-25T11:30:12.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:12.255+0000 7f5bb59dc640 1 -- 172.21.15.136:0/1866990676 wait complete. 2024-04-25T11:30:12.265 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:14.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:14.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 Processor -- start 2024-04-25T11:30:14.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 -- start start 2024-04-25T11:30:14.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5514154060 0x7f55140c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f551415aa00 0x7f5514164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55140c8d40 con 0x7f551415a5f0 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f55140c8fd0 con 0x7f5514154060 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551abbe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55140c3880 con 0x7f551415aa00 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5514154060 0x7f55140c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5514154060 0x7f55140c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52142/0 (socket says 172.21.15.136:52142) 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 -- 172.21.15.136:0/4102296251 learned_addr learned my addr 172.21.15.136:0/4102296251 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551915b640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f551415aa00 0x7f5514164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 -- 172.21.15.136:0/4102296251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f551415aa00 msgr2=0x7f5514164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f551415aa00 0x7f5514164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 -- 172.21.15.136:0/4102296251 >> v1:172.21.15.157:6789/0 conn(0x7f551415a5f0 legacy=0x7f55140c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:14.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.431+0000 7f551895a640 1 -- 172.21.15.136:0/4102296251 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551406d9b0 con 0x7f5514154060 2024-04-25T11:30:14.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551895a640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5514154060 0x7f55140c8800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5510009e50 tx=0x7f551001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592bfa0b47acd43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:14.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f550b7fe640 1 -- 172.21.15.136:0/4102296251 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55100b4070 con 0x7f5514154060 2024-04-25T11:30:14.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f550b7fe640 1 -- 172.21.15.136:0/4102296251 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55100af400 con 0x7f5514154060 2024-04-25T11:30:14.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f550b7fe640 1 -- 172.21.15.136:0/4102296251 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55100af7d0 con 0x7f5514154060 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 -- 172.21.15.136:0/4102296251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5514154060 msgr2=0x7f55140c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5514154060 0x7f55140c8800 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5510009e50 tx=0x7f551001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 -- 172.21.15.136:0/4102296251 shutdown_connections 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f551415aa00 0x7f5514164700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 --2- 172.21.15.136:0/4102296251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5514154060 0x7f55140c8800 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 -- 172.21.15.136:0/4102296251 >> 172.21.15.136:0/4102296251 conn(0x7f55140ca5a0 msgr2=0x7f55140ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:14.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 -- 172.21.15.136:0/4102296251 shutdown_connections 2024-04-25T11:30:14.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.435+0000 7f551abbe640 1 -- 172.21.15.136:0/4102296251 wait complete. 2024-04-25T11:30:14.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 Processor -- start 2024-04-25T11:30:14.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 -- start start 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5514154060 0x7f55141e4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 0x7f551414b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f551406df30 con 0x7f551415a5f0 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5514154470 con 0x7f551415aa00 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551abbe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f551415a0d0 con 0x7f5514154060 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52148/0 (socket says 172.21.15.136:52148) 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 -- 172.21.15.136:0/926616262 learned_addr learned my addr 172.21.15.136:0/926616262 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551895a640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5514154060 0x7f55141e4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551915b640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 0x7f551414b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 -- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5514154060 msgr2=0x7f55141e4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5514154060 0x7f55141e4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 -- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 msgr2=0x7f551414b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 0x7f551414b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 -- 172.21.15.136:0/926616262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55141e5090 con 0x7f551415a5f0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f551915b640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 0x7f551414b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f550bfff640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f550400b3c0 tx=0x7f550400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504011830 con 0x7f551415a5f0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5504016070 con 0x7f551415a5f0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.439+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504011bf0 con 0x7f551415a5f0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55141e3960 con 0x7f551415a5f0 2024-04-25T11:30:14.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55141e52a0 con 0x7f551415a5f0 2024-04-25T11:30:14.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551406a350 con 0x7f551415a5f0 2024-04-25T11:30:14.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f550401b460 con 0x7f551415a5f0 2024-04-25T11:30:14.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f55097fa640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 0x7f5500071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:14.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f550408f250 con 0x7f551415a5f0 2024-04-25T11:30:14.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f551895a640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 0x7f5500071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:14.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.443+0000 7f551895a640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 0x7f5500071910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55100b1000 tx=0x7f55100aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:14.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.451+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5504020020 con 0x7f551415a5f0 2024-04-25T11:30:14.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f551415e210 con 0x7f551415a5f0 2024-04-25T11:30:14.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f55097fa640 1 -- 172.21.15.136:0/926616262 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6095 (secure 0 0 0) 0x7f550405bc70 con 0x7f551415a5f0 2024-04-25T11:30:14.692 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:14.693 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:13.892622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:resolve","state_seq":21,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:14.693 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 39 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 msgr2=0x7f5500071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 0x7f5500071910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55100b1000 tx=0x7f55100aa000 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 msgr2=0x7f5514074400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f550400b3c0 tx=0x7f550400b880 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 shutdown_connections 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f551415aa00 0x7f551414b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551415a5f0 0x7f5514074400 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f550006f400 0x7f5500071910 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 --2- 172.21.15.136:0/926616262 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5514154060 0x7f55141e4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 >> 172.21.15.136:0/926616262 conn(0x7f55140ca5a0 msgr2=0x7f551414d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 shutdown_connections 2024-04-25T11:30:14.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:14.703+0000 7f551abbe640 1 -- 172.21.15.136:0/926616262 wait complete. 2024-04-25T11:30:14.718 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:16.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.911+0000 7f41d872a640 1 Processor -- start 2024-04-25T11:30:16.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.911+0000 7f41d872a640 1 -- start start 2024-04-25T11:30:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d872a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41d015a5f0 0x7f41d00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d872a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015aa00 0x7f41d00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d872a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d00c3880 con 0x7f41d0154060 2024-04-25T11:30:16.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d872a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f41d00c8fd0 con 0x7f41d015a5f0 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d872a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41d006d9b0 con 0x7f41d015aa00 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41d015a5f0 0x7f41d00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d64c6640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f41d0154060 0x7f41d00c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:51938/0 (socket says 172.21.15.136:51938) 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41d015a5f0 0x7f41d00c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33642/0 (socket says 172.21.15.136:33642) 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d64c6640 1 -- 172.21.15.136:0/3685523692 learned_addr learned my addr 172.21.15.136:0/3685523692 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d6cc7640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015aa00 0x7f41d00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 -- 172.21.15.136:0/3685523692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015aa00 msgr2=0x7f41d00c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015aa00 0x7f41d00c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 -- 172.21.15.136:0/3685523692 >> v1:172.21.15.136:6789/0 conn(0x7f41d0154060 legacy=0x7f41d00c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 -- 172.21.15.136:0/3685523692 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d0154470 con 0x7f41d015a5f0 2024-04-25T11:30:16.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d54c4640 1 -- 172.21.15.136:0/3685523692 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294548376 0 0) 0x7f41d00c3880 con 0x7f41d0154060 2024-04-25T11:30:16.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d5cc5640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015a5f0 0x7f41d00c71b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f41d0154640 tx=0x7f41c001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fac542116df4164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:16.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d54c4640 1 -- 172.21.15.136:0/3685523692 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c00b5070 con 0x7f41d015a5f0 2024-04-25T11:30:16.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d54c4640 1 -- 172.21.15.136:0/3685523692 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c00a8990 con 0x7f41d015a5f0 2024-04-25T11:30:16.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.915+0000 7f41d54c4640 1 -- 172.21.15.136:0/3685523692 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c00a8d60 con 0x7f41d015a5f0 2024-04-25T11:30:16.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- 172.21.15.136:0/3685523692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015a5f0 msgr2=0x7f41d00c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:16.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015a5f0 0x7f41d00c71b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f41d0154640 tx=0x7f41c001ec90 comp rx=0 tx=0).stop 2024-04-25T11:30:16.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- 172.21.15.136:0/3685523692 shutdown_connections 2024-04-25T11:30:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015aa00 0x7f41d00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 --2- 172.21.15.136:0/3685523692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015a5f0 0x7f41d00c71b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- 172.21.15.136:0/3685523692 >> 172.21.15.136:0/3685523692 conn(0x7f41d00ca5a0 msgr2=0x7f41d00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- 172.21.15.136:0/3685523692 shutdown_connections 2024-04-25T11:30:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- 172.21.15.136:0/3685523692 wait complete. 2024-04-25T11:30:16.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 Processor -- start 2024-04-25T11:30:16.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 -- start start 2024-04-25T11:30:16.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.919+0000 7f41d872a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41d0154060 0x7f41d01edc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015a5f0 0x7f41d014d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f41d015a0d0 con 0x7f41d015aa00 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41d006df30 con 0x7f41d0154060 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41d0159d80 con 0x7f41d015a5f0 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33646/0 (socket says 172.21.15.136:33646) 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 -- 172.21.15.136:0/3892163519 learned_addr learned my addr 172.21.15.136:0/3892163519 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d64c6640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41d0154060 0x7f41d01edc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d5cc5640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015a5f0 0x7f41d014d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 -- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015a5f0 msgr2=0x7f41d014d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015a5f0 0x7f41d014d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 -- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41d0154060 msgr2=0x7f41d01edc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41d0154060 0x7f41d01edc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 -- 172.21.15.136:0/3892163519 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d0155a90 con 0x7f41d015aa00 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d6cc7640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f41c8007d80 tx=0x7f41c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8014420 con 0x7f41d015aa00 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c8019070 con 0x7f41d015aa00 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d0074e50 con 0x7f41d015aa00 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8014800 con 0x7f41d015aa00 2024-04-25T11:30:16.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d01ee180 con 0x7f41d015aa00 2024-04-25T11:30:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41d006a350 con 0x7f41d015aa00 2024-04-25T11:30:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f41c80149a0 con 0x7f41d015aa00 2024-04-25T11:30:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 0x7f41b4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f41c8090060 con 0x7f41d015aa00 2024-04-25T11:30:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.923+0000 7f41d64c6640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 0x7f41b4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:16.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.927+0000 7f41d64c6640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 0x7f41b4071910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f41cc000f70 tx=0x7f41cc007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:16.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:16.931+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f41c805ca80 con 0x7f41d015aa00 2024-04-25T11:30:17.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.151+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41d00c85e0 con 0x7f41d015aa00 2024-04-25T11:30:17.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.151+0000 7f41c6ffd640 1 -- 172.21.15.136:0/3892163519 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6095 (secure 0 0 0) 0x7f41c805c380 con 0x7f41d015aa00 2024-04-25T11:30:17.142 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:17.142 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:13.892622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:resolve","state_seq":21,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:17.142 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 39 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 msgr2=0x7f41b4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 0x7f41b4071910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f41cc000f70 tx=0x7f41cc007c10 comp rx=0 tx=0).stop 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 msgr2=0x7f41d00cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f41c8007d80 tx=0x7f41c800e4b0 comp rx=0 tx=0).stop 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 shutdown_connections 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41d015aa00 0x7f41d00cc8f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f41b406f400 0x7f41b4071910 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f41cc000f70 tx=0x7f41cc007c10 comp rx=0 tx=0).stop 2024-04-25T11:30:17.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41d015a5f0 0x7f41d014d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:17.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 --2- 172.21.15.136:0/3892163519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41d0154060 0x7f41d01edc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:17.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 >> 172.21.15.136:0/3892163519 conn(0x7f41d00ca5a0 msgr2=0x7f41d00c2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:17.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 shutdown_connections 2024-04-25T11:30:17.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:17.155+0000 7f41d872a640 1 -- 172.21.15.136:0/3892163519 wait complete. 2024-04-25T11:30:17.164 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:19.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:19.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 Processor -- start 2024-04-25T11:30:19.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 -- start start 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2d580c96e0 0x7f2d581540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d5815aae0 0x7f2d580c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d580c9af0 con 0x7f2d580c79e0 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2d5806cc10 con 0x7f2d580c96e0 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.355+0000 7f2d5cc62640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d580cb220 con 0x7f2d5815aae0 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2d580c96e0 0x7f2d581540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2d580c96e0 0x7f2d581540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33650/0 (socket says 172.21.15.136:33650) 2024-04-25T11:30:19.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 -- 172.21.15.136:0/332682573 learned_addr learned my addr 172.21.15.136:0/332682573 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d56d9d640 1 --2- 172.21.15.136:0/332682573 >> v2:172.21.15.157:3300/0 conn(0x7f2d5815aae0 0x7f2d580c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 -- 172.21.15.136:0/332682573 >> v2:172.21.15.157:3300/0 conn(0x7f2d5815aae0 msgr2=0x7f2d580c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 --2- 172.21.15.136:0/332682573 >> v2:172.21.15.157:3300/0 conn(0x7f2d5815aae0 0x7f2d580c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 -- 172.21.15.136:0/332682573 >> v1:172.21.15.157:6789/0 conn(0x7f2d580c79e0 legacy=0x7f2d5815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 -- 172.21.15.136:0/332682573 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d580cb500 con 0x7f2d580c96e0 2024-04-25T11:30:19.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5659c640 1 --2- 172.21.15.136:0/332682573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d580c96e0 0x7f2d581540d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2d58075140 tx=0x7f2d4801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4904c9888eaf3fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:19.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5559a640 1 -- 172.21.15.136:0/332682573 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d480b5070 con 0x7f2d580c96e0 2024-04-25T11:30:19.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5559a640 1 -- 172.21.15.136:0/332682573 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d480a89d0 con 0x7f2d580c96e0 2024-04-25T11:30:19.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5559a640 1 -- 172.21.15.136:0/332682573 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d480a8da0 con 0x7f2d580c96e0 2024-04-25T11:30:19.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 -- 172.21.15.136:0/332682573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d580c96e0 msgr2=0x7f2d581540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:19.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/332682573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d580c96e0 0x7f2d581540d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2d58075140 tx=0x7f2d4801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 -- 172.21.15.136:0/332682573 shutdown_connections 2024-04-25T11:30:19.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/332682573 >> v2:172.21.15.157:3300/0 conn(0x7f2d5815aae0 0x7f2d580c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/332682573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d580c96e0 0x7f2d581540d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 -- 172.21.15.136:0/332682573 >> 172.21.15.136:0/332682573 conn(0x7f2d5806f9d0 msgr2=0x7f2d5806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:19.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 -- 172.21.15.136:0/332682573 shutdown_connections 2024-04-25T11:30:19.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.359+0000 7f2d5cc62640 1 -- 172.21.15.136:0/332682573 wait complete. 2024-04-25T11:30:19.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.363+0000 7f2d5cc62640 1 Processor -- start 2024-04-25T11:30:19.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.363+0000 7f2d5cc62640 1 -- start start 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.363+0000 7f2d5cc62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 0x7f2d580cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 0x7f2d58137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2d580cf0f0 con 0x7f2d5815aae0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d58154610 con 0x7f2d580c96e0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d580ced40 con 0x7f2d580c79e0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33660/0 (socket says 172.21.15.136:33660) 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 -- 172.21.15.136:0/438697504 learned_addr learned my addr 172.21.15.136:0/438697504 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5659c640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 0x7f2d580cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d55d9b640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 0x7f2d58137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 -- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 msgr2=0x7f2d580cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 0x7f2d580cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 -- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 msgr2=0x7f2d58137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 0x7f2d58137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 -- 172.21.15.136:0/438697504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d580ccdf0 con 0x7f2d5815aae0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5659c640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 0x7f2d580cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d55d9b640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 0x7f2d58137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d56d9d640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000cc60 tx=0x7f2d5000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d500039a0 con 0x7f2d5815aae0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d50003b40 con 0x7f2d5815aae0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d580cb9a0 con 0x7f2d5815aae0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50004020 con 0x7f2d5815aae0 2024-04-25T11:30:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5813cbe0 con 0x7f2d5815aae0 2024-04-25T11:30:19.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5806a350 con 0x7f2d5815aae0 2024-04-25T11:30:19.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.367+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2d500041c0 con 0x7f2d5815aae0 2024-04-25T11:30:19.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.371+0000 7f2d46ffd640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 0x7f2d38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:19.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.371+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f2d5008fc50 con 0x7f2d5815aae0 2024-04-25T11:30:19.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.371+0000 7f2d5659c640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 0x7f2d38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:19.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.371+0000 7f2d5659c640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 0x7f2d38071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d480b2000 tx=0x7f2d48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:19.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.375+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2d50095020 con 0x7f2d5815aae0 2024-04-25T11:30:19.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.607+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d581579a0 con 0x7f2d5815aae0 2024-04-25T11:30:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d46ffd640 1 -- 172.21.15.136:0/438697504 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6095 (secure 0 0 0) 0x7f2d50014030 con 0x7f2d5815aae0 2024-04-25T11:30:19.600 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:19.600 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:13.892622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:resolve","state_seq":21,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:19.600 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 39 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 msgr2=0x7f2d38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 0x7f2d38071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d480b2000 tx=0x7f2d48008040 comp rx=0 tx=0).stop 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 msgr2=0x7f2d58137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000cc60 tx=0x7f2d5000a5c0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 shutdown_connections 2024-04-25T11:30:19.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d5815aae0 0x7f2d58137dc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d580c96e0 0x7f2d58137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3806f400 0x7f2d38071910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 --2- 172.21.15.136:0/438697504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d580c79e0 0x7f2d580cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 >> 172.21.15.136:0/438697504 conn(0x7f2d5806f9d0 msgr2=0x7f2d581534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 shutdown_connections 2024-04-25T11:30:19.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:19.611+0000 7f2d5cc62640 1 -- 172.21.15.136:0/438697504 wait complete. 2024-04-25T11:30:19.621 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:21.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:21.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 Processor -- start 2024-04-25T11:30:21.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 -- start start 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f612815ff60 0x7f61280ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6128160370 0x7f61280ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612815c6f0 con 0x7f612815c0d0 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f61280c6310 con 0x7f612815ff60 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612f30d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f612815c990 con 0x7f6128160370 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612c8a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f612815ff60 0x7f61280ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612d0a9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f612815c0d0 0x7f61280cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:51944/0 (socket says 172.21.15.136:51944) 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612c8a8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f612815ff60 0x7f61280ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33670/0 (socket says 172.21.15.136:33670) 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.803+0000 7f612d0a9640 1 -- 172.21.15.136:0/437817233 learned_addr learned my addr 172.21.15.136:0/437817233 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612d8aa640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6128160370 0x7f61280ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612c8a8640 1 -- 172.21.15.136:0/437817233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6128160370 msgr2=0x7f61280ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612c8a8640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6128160370 0x7f61280ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612c8a8640 1 -- 172.21.15.136:0/437817233 >> v1:172.21.15.136:6789/0 conn(0x7f612815c0d0 legacy=0x7f61280cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612c8a8640 1 -- 172.21.15.136:0/437817233 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612815cc30 con 0x7f612815ff60 2024-04-25T11:30:21.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f611ffff640 1 -- 172.21.15.136:0/437817233 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757595629 0 0) 0x7f612815c6f0 con 0x7f612815c0d0 2024-04-25T11:30:21.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612c8a8640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f612815ff60 0x7f61280ce260 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f611801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f3d79981d19d07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:21.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f611ffff640 1 -- 172.21.15.136:0/437817233 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61180b2070 con 0x7f612815ff60 2024-04-25T11:30:21.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f611ffff640 1 -- 172.21.15.136:0/437817233 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61180ad640 con 0x7f612815ff60 2024-04-25T11:30:21.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f611ffff640 1 -- 172.21.15.136:0/437817233 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61180ada10 con 0x7f612815ff60 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 -- 172.21.15.136:0/437817233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f612815ff60 msgr2=0x7f61280ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f612815ff60 0x7f61280ce260 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f61180097c0 tx=0x7f611801c4a0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 -- 172.21.15.136:0/437817233 shutdown_connections 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6128160370 0x7f61280ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 --2- 172.21.15.136:0/437817233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f612815ff60 0x7f61280ce260 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.807+0000 7f612f30d640 1 -- 172.21.15.136:0/437817233 >> 172.21.15.136:0/437817233 conn(0x7f612806f9d0 msgr2=0x7f612806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.811+0000 7f612f30d640 1 -- 172.21.15.136:0/437817233 shutdown_connections 2024-04-25T11:30:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.811+0000 7f612f30d640 1 -- 172.21.15.136:0/437817233 wait complete. 2024-04-25T11:30:21.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.811+0000 7f612f30d640 1 Processor -- start 2024-04-25T11:30:21.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.811+0000 7f612f30d640 1 -- start start 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f612815c0d0 0x7f6128156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 0x7f612815b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f61280c9170 con 0x7f6128160370 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f61280cabc0 con 0x7f612815ff60 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6128157500 con 0x7f612815c0d0 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33672/0 (socket says 172.21.15.136:33672) 2024-04-25T11:30:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 -- 172.21.15.136:0/1916183110 learned_addr learned my addr 172.21.15.136:0/1916183110 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d0a9640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f612815c0d0 0x7f6128156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612c8a8640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 0x7f612815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 -- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f612815c0d0 msgr2=0x7f6128156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f612815c0d0 0x7f6128156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 -- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 msgr2=0x7f612815b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 0x7f612815b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 -- 172.21.15.136:0/1916183110 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61281f5300 con 0x7f6128160370 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612c8a8640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 0x7f612815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d8aa640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6120007d70 tx=0x7f612000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120014420 con 0x7f6128160370 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120019070 con 0x7f6128160370 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61281534b0 con 0x7f6128160370 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120014820 con 0x7f6128160370 2024-04-25T11:30:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61281f58a0 con 0x7f6128160370 2024-04-25T11:30:21.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61080052a0 con 0x7f6128160370 2024-04-25T11:30:21.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f61200149c0 con 0x7f6128160370 2024-04-25T11:30:21.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 0x7f610c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:21.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f612008fec0 con 0x7f6128160370 2024-04-25T11:30:21.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.815+0000 7f612d0a9640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 0x7f610c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:21.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.819+0000 7f612d0a9640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 0x7f610c071940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6124000f70 tx=0x7f6124007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:21.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:21.823+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f612005c970 con 0x7f6128160370 2024-04-25T11:30:22.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.063+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6108005860 con 0x7f6128160370 2024-04-25T11:30:22.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.063+0000 7f611dffb640 1 -- 172.21.15.136:0/1916183110 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6097 (secure 0 0 0) 0x7f612005c270 con 0x7f6128160370 2024-04-25T11:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:19.319209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:30:22.054 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 40 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 msgr2=0x7f610c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 0x7f610c071940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6124000f70 tx=0x7f6124007c10 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 msgr2=0x7f6128152780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6120007d70 tx=0x7f612000e4b0 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 shutdown_connections 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6128160370 0x7f6128152780 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f612815ff60 0x7f612815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f610c06f430 0x7f610c071940 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 --2- 172.21.15.136:0/1916183110 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f612815c0d0 0x7f6128156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:22.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 >> 172.21.15.136:0/1916183110 conn(0x7f612806f9d0 msgr2=0x7f61280c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:22.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 shutdown_connections 2024-04-25T11:30:22.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:22.067+0000 7f612f30d640 1 -- 172.21.15.136:0/1916183110 wait complete. 2024-04-25T11:30:22.075 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4297": { "addr": "172.21.15.157:6836/694883637", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 694883637, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 694883637, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4297, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 12 }, "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:30:19.319209+0000", "qdb_cluster": [ 4297, 4303, 5238, 5385 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 4297, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:30:22.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:30:22.110 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:30:24.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 Processor -- start 2024-04-25T11:30:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 -- start start 2024-04-25T11:30:24.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86380c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f863815a5f0 0x7f86380c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86380c8d40 con 0x7f863815aa00 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f86380c8fd0 con 0x7f863815a5f0 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863f3bd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f86380c3880 con 0x7f8638154060 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f863815a5f0 0x7f86380c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f863815a5f0 0x7f86380c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33684/0 (socket says 172.21.15.136:33684) 2024-04-25T11:30:24.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863d95a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f863815aa00 0x7f863815d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:51950/0 (socket says 172.21.15.136:51950) 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 -- 172.21.15.136:0/864848196 learned_addr learned my addr 172.21.15.136:0/864848196 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863d159640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86380c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 -- 172.21.15.136:0/864848196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 msgr2=0x7f86380c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86380c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 -- 172.21.15.136:0/864848196 >> v1:172.21.15.136:6789/0 conn(0x7f863815aa00 legacy=0x7f863815d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f863c958640 1 -- 172.21.15.136:0/864848196 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863806d9b0 con 0x7f863815a5f0 2024-04-25T11:30:24.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.263+0000 7f862ffff640 1 -- 172.21.15.136:0/864848196 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3127372697 0 0) 0x7f86380c8d40 con 0x7f863815aa00 2024-04-25T11:30:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863c958640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815a5f0 0x7f86380c71b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f863806db80 tx=0x7f862801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599f67f07a457f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f862ffff640 1 -- 172.21.15.136:0/864848196 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86280b5070 con 0x7f863815a5f0 2024-04-25T11:30:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f862ffff640 1 -- 172.21.15.136:0/864848196 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86280a8a30 con 0x7f863815a5f0 2024-04-25T11:30:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f862ffff640 1 -- 172.21.15.136:0/864848196 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86280a8e00 con 0x7f863815a5f0 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 -- 172.21.15.136:0/864848196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815a5f0 msgr2=0x7f86380c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815a5f0 0x7f86380c71b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f863806db80 tx=0x7f862801ed30 comp rx=0 tx=0).stop 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 -- 172.21.15.136:0/864848196 shutdown_connections 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815a5f0 0x7f86380c71b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 --2- 172.21.15.136:0/864848196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86380c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 -- 172.21.15.136:0/864848196 >> 172.21.15.136:0/864848196 conn(0x7f86380ca5a0 msgr2=0x7f86380ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:24.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 -- 172.21.15.136:0/864848196 shutdown_connections 2024-04-25T11:30:24.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.267+0000 7f863f3bd640 1 -- 172.21.15.136:0/864848196 wait complete. 2024-04-25T11:30:24.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 Processor -- start 2024-04-25T11:30:24.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- start start 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86381f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f863815a5f0 0x7f863814b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f863806df30 con 0x7f863815aa00 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8638154470 con 0x7f863815a5f0 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f863815a0d0 con 0x7f8638154060 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33686/0 (socket says 172.21.15.136:33686) 2024-04-25T11:30:24.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 -- 172.21.15.136:0/2658250128 learned_addr learned my addr 172.21.15.136:0/2658250128 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:24.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d159640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86381f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863c958640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f863815a5f0 0x7f863814b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 -- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 msgr2=0x7f86381f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86381f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 -- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f863815a5f0 msgr2=0x7f863814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f863815a5f0 0x7f863814b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 -- 172.21.15.136:0/2658250128 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8638075a90 con 0x7f863815aa00 2024-04-25T11:30:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863d95a640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f863000c700 tx=0x7f863000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:24.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630018070 con 0x7f863815aa00 2024-04-25T11:30:24.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f863000ce10 con 0x7f863815aa00 2024-04-25T11:30:24.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86381f0f00 con 0x7f863815aa00 2024-04-25T11:30:24.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863000a880 con 0x7f863815aa00 2024-04-25T11:30:24.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.271+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f863815b0c0 con 0x7f863815aa00 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f863806a350 con 0x7f863815aa00 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f863001b070 con 0x7f863815aa00 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f862dffb640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 0x7f861c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f863008edb0 con 0x7f863815aa00 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f863d159640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 0x7f861c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.275+0000 7f863d159640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 0x7f861c071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f863400b930 tx=0x7f8634008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:24.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.283+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8638075a90 con 0x7f863815aa00 2024-04-25T11:30:24.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8638157240 con 0x7f863815aa00 2024-04-25T11:30:24.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f862dffb640 1 -- 172.21.15.136:0/2658250128 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6097 (secure 0 0 0) 0x7f863005b740 con 0x7f863815aa00 2024-04-25T11:30:24.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:24.500 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:19.319209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385]},"id":1}]} 2024-04-25T11:30:24.500 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 40 2024-04-25T11:30:24.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 msgr2=0x7f861c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 0x7f861c071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f863400b930 tx=0x7f8634008040 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 msgr2=0x7f863814b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f863000c700 tx=0x7f863000cbc0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 shutdown_connections 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f863815aa00 0x7f863814b860 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f863815a5f0 0x7f863814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f861c06f400 0x7f861c071910 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 --2- 172.21.15.136:0/2658250128 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8638154060 0x7f86381f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 >> 172.21.15.136:0/2658250128 conn(0x7f86380ca5a0 msgr2=0x7f863814d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:24.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 shutdown_connections 2024-04-25T11:30:24.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:24.511+0000 7f863f3bd640 1 -- 172.21.15.136:0/2658250128 wait complete. 2024-04-25T11:30:24.522 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:26.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.679+0000 7fe22a897640 1 Processor -- start 2024-04-25T11:30:26.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.679+0000 7fe22a897640 1 -- start start 2024-04-25T11:30:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe22a897640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe22406d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe22a897640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2240b4480 con 0x7fe22415c480 2024-04-25T11:30:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe22a897640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe22415c990 con 0x7fe2240b5810 2024-04-25T11:30:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe22a897640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2240b4a90 con 0x7fe22415c070 2024-04-25T11:30:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe22415c480 0x7fe224166910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52356/0 (socket says 172.21.15.136:52356) 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 -- 172.21.15.136:0/3476606192 learned_addr learned my addr 172.21.15.136:0/3476606192 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe2237fe640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe22406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981943295 0 0) 0x7fe2240b4480 con 0x7fe22415c480 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe20c003530 con 0x7fe22415c480 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3165383161 0 0) 0x7fe20c003530 con 0x7fe22415c480 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 msgr2=0x7fe22406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe22406d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 >> v1:172.21.15.157:6789/0 conn(0x7fe2240b5810 legacy=0x7fe2240b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22415cc70 con 0x7fe22415c480 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fe21c001a00 con 0x7fe22415c480 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 >> v1:172.21.15.136:6789/0 conn(0x7fe22415c480 legacy=0x7fe224166910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c0039f0 0x7fe20c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe20c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe2240b4a90 con 0x7fe20c004d50 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c00bb90 con 0x7fe20c0039f0 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe20c00bd90 con 0x7fe22415c070 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe223fff640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c0039f0 0x7fe20c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe2237fe640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe20c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 -- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 msgr2=0x7fe20c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe22415c070 0x7fe20c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 -- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c0039f0 msgr2=0x7fe20c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe20c0039f0 0x7fe20c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.683+0000 7fe228e34640 1 -- 172.21.15.136:0/3476606192 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c003530 con 0x7fe20c004d50 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe228e34640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00a850 tx=0x7fe21c01d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e575f3702fae6be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00a4a0 con 0x7fe20c004d50 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe21c0ad650 con 0x7fe20c004d50 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe222ffd640 1 -- 172.21.15.136:0/3476606192 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0ada40 con 0x7fe20c004d50 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 -- 172.21.15.136:0/3476606192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe20c004d50 msgr2=0x7fe20c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 --2- 172.21.15.136:0/3476606192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00a850 tx=0x7fe21c01d3a0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe228e34640 1 -- 172.21.15.136:0/3476606192 reap_dead start 2024-04-25T11:30:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 -- 172.21.15.136:0/3476606192 shutdown_connections 2024-04-25T11:30:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 -- 172.21.15.136:0/3476606192 >> 172.21.15.136:0/3476606192 conn(0x7fe2240743a0 msgr2=0x7fe2240d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 -- 172.21.15.136:0/3476606192 shutdown_connections 2024-04-25T11:30:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.687+0000 7fe22a897640 1 -- 172.21.15.136:0/3476606192 wait complete. 2024-04-25T11:30:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 Processor -- start 2024-04-25T11:30:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- start start 2024-04-25T11:30:26.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe224150310 0x7fe2241f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe224155440 0x7fe224155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe22406e030 con 0x7fe2240ca440 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe22415cf50 con 0x7fe224150310 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2240cbee0 con 0x7fe224155440 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44848/0 (socket says 172.21.15.136:44848) 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 -- 172.21.15.136:0/1159102177 learned_addr learned my addr 172.21.15.136:0/1159102177 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe2237fe640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe224150310 0x7fe2241f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe228e34640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe224155440 0x7fe224155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 -- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe224155440 msgr2=0x7fe224155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe224155440 0x7fe224155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 -- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe224150310 msgr2=0x7fe2241f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe224150310 0x7fe2241f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 -- 172.21.15.136:0/1159102177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2241602c0 con 0x7fe2240ca440 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe223fff640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe21400e4e0 tx=0x7fe21400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140052e0 con 0x7fe2240ca440 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe214018070 con 0x7fe2240ca440 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2241f2e70 con 0x7fe2240ca440 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe214003820 con 0x7fe2240ca440 2024-04-25T11:30:26.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2240c9000 con 0x7fe2240ca440 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.691+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe22406a350 con 0x7fe2240ca440 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.695+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe2140039c0 con 0x7fe2240ca440 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.695+0000 7fe220ff9640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 0x7fe210071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.695+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7fe21408f3a0 con 0x7fe2240ca440 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.695+0000 7fe2237fe640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 0x7fe210071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.695+0000 7fe2237fe640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 0x7fe210071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe218009a30 tx=0x7fe218008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:26.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.699+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe2241f2e70 con 0x7fe2240ca440 2024-04-25T11:30:26.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2240b4140 con 0x7fe2240ca440 2024-04-25T11:30:26.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe220ff9640 1 -- 172.21.15.136:0/1159102177 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6094 (secure 0 0 0) 0x7fe2241602c0 con 0x7fe2240ca440 2024-04-25T11:30:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:26.923 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:25.095588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:26.923 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 41 2024-04-25T11:30:26.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 msgr2=0x7fe210071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 0x7fe210071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe218009a30 tx=0x7fe218008040 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 msgr2=0x7fe2240caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe21400e4e0 tx=0x7fe21400e9a0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 shutdown_connections 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe21006f400 0x7fe210071910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe224155440 0x7fe224155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe224150310 0x7fe2241f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 --2- 172.21.15.136:0/1159102177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2240ca440 0x7fe2240caec0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 >> 172.21.15.136:0/1159102177 conn(0x7fe2240743a0 msgr2=0x7fe22415ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 shutdown_connections 2024-04-25T11:30:26.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:26.935+0000 7fe22a897640 1 -- 172.21.15.136:0/1159102177 wait complete. 2024-04-25T11:30:26.947 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:28.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.103+0000 7fc134db5640 1 Processor -- start 2024-04-25T11:30:29.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.103+0000 7fc134db5640 1 -- start start 2024-04-25T11:30:29.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc134db5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc1300c96e0 0x7fc1301540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc134db5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc1300c79e0 0x7fc1300c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc134db5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1300c9af0 con 0x7fc13015a3d0 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc134db5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc13006cc10 con 0x7fc1300c96e0 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc134db5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc1300cb220 con 0x7fc1300c79e0 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12ed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc13015a3d0 0x7fc13015a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52364/0 (socket says 172.21.15.136:52364) 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12ed9d640 1 -- 172.21.15.136:0/1422013975 learned_addr learned my addr 172.21.15.136:0/1422013975 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:29.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12e59c640 1 --2- 172.21.15.136:0/1422013975 >> v2:172.21.15.136:3300/0 conn(0x7fc1300c96e0 0x7fc1301540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12dd9b640 1 --2- 172.21.15.136:0/1422013975 >> v2:172.21.15.157:3300/0 conn(0x7fc1300c79e0 0x7fc1300c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12e59c640 1 -- 172.21.15.136:0/1422013975 >> v2:172.21.15.157:3300/0 conn(0x7fc1300c79e0 msgr2=0x7fc1300c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:29.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12e59c640 1 --2- 172.21.15.136:0/1422013975 >> v2:172.21.15.157:3300/0 conn(0x7fc1300c79e0 0x7fc1300c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12e59c640 1 -- 172.21.15.136:0/1422013975 >> v1:172.21.15.136:6789/0 conn(0x7fc13015a3d0 legacy=0x7fc13015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.107+0000 7fc12e59c640 1 -- 172.21.15.136:0/1422013975 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1300cb500 con 0x7fc1300c96e0 2024-04-25T11:30:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc12e59c640 1 --2- 172.21.15.136:0/1422013975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc1301540d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc1300c8990 tx=0x7fc11801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19c7d304b4e72d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc12d59a640 1 -- 172.21.15.136:0/1422013975 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180b0070 con 0x7fc1300c96e0 2024-04-25T11:30:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc12d59a640 1 -- 172.21.15.136:0/1422013975 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1180ab600 con 0x7fc1300c96e0 2024-04-25T11:30:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc12d59a640 1 -- 172.21.15.136:0/1422013975 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180ab9d0 con 0x7fc1300c96e0 2024-04-25T11:30:29.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 -- 172.21.15.136:0/1422013975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 msgr2=0x7fc1301540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 --2- 172.21.15.136:0/1422013975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc1301540d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc1300c8990 tx=0x7fc11801c3a0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 -- 172.21.15.136:0/1422013975 shutdown_connections 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 --2- 172.21.15.136:0/1422013975 >> v2:172.21.15.157:3300/0 conn(0x7fc1300c79e0 0x7fc1300c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 --2- 172.21.15.136:0/1422013975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc1301540d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 -- 172.21.15.136:0/1422013975 >> 172.21.15.136:0/1422013975 conn(0x7fc13006f9d0 msgr2=0x7fc13006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 -- 172.21.15.136:0/1422013975 shutdown_connections 2024-04-25T11:30:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.111+0000 7fc134db5640 1 -- 172.21.15.136:0/1422013975 wait complete. 2024-04-25T11:30:29.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 Processor -- start 2024-04-25T11:30:29.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 -- start start 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1300c79e0 0x7fc1301f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 0x7fc130150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc1300cf0f0 con 0x7fc1300c96e0 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc130154610 con 0x7fc13015a3d0 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc134db5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc1300ced40 con 0x7fc1300c79e0 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc12dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc12dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44864/0 (socket says 172.21.15.136:44864) 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc12dd9b640 1 -- 172.21.15.136:0/2602928406 learned_addr learned my addr 172.21.15.136:0/2602928406 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc12e59c640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1300c79e0 0x7fc1301f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.115+0000 7fc12ed9d640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 0x7fc130150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 -- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1300c79e0 msgr2=0x7fc1301f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1300c79e0 0x7fc1301f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 -- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 msgr2=0x7fc130150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 0x7fc130150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 -- 172.21.15.136:0/2602928406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1300cd8b0 con 0x7fc1300c96e0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12ed9d640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 0x7fc130150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12dd9b640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc120007e90 tx=0x7fc12000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120014620 con 0x7fc1300c96e0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc120019070 con 0x7fc1300c96e0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1301595a0 con 0x7fc1300c96e0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc120014a20 con 0x7fc1300c96e0 2024-04-25T11:30:29.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13015c170 con 0x7fc1300c96e0 2024-04-25T11:30:29.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc13006a350 con 0x7fc1300c96e0 2024-04-25T11:30:29.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc120014bc0 con 0x7fc1300c96e0 2024-04-25T11:30:29.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 0x7fc110071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:29.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7fc120090070 con 0x7fc1300c96e0 2024-04-25T11:30:29.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.119+0000 7fc12e59c640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 0x7fc110071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:29.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.123+0000 7fc12e59c640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 0x7fc110071940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc130075140 tx=0x7fc1180ad040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:29.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.127+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc12005ca90 con 0x7fc1300c96e0 2024-04-25T11:30:29.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.335+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc130159da0 con 0x7fc1300c96e0 2024-04-25T11:30:29.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.335+0000 7fc126ffd640 1 -- 172.21.15.136:0/2602928406 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6094 (secure 0 0 0) 0x7fc12005c390 con 0x7fc1300c96e0 2024-04-25T11:30:29.324 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:29.324 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:25.095588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:rejoin","state_seq":25,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:29.325 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 41 2024-04-25T11:30:29.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 msgr2=0x7fc110071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 0x7fc110071940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc130075140 tx=0x7fc1180ad040 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 msgr2=0x7fc130149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc120007e90 tx=0x7fc12000e4b0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 shutdown_connections 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc13015a3d0 0x7fc130150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1300c96e0 0x7fc130149e30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc11006f430 0x7fc110071940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 --2- 172.21.15.136:0/2602928406 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1300c79e0 0x7fc1301f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 >> 172.21.15.136:0/2602928406 conn(0x7fc13006f9d0 msgr2=0x7fc1300c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 shutdown_connections 2024-04-25T11:30:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:29.339+0000 7fc134db5640 1 -- 172.21.15.136:0/2602928406 wait complete. 2024-04-25T11:30:29.345 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:31.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:31.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 Processor -- start 2024-04-25T11:30:31.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 -- start start 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f052c15ff60 0x7f052c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052c15c6f0 con 0x7f052c15c0d0 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052c15c990 con 0x7f052c160370 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053336f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f052c0c6310 con 0x7f052c15ff60 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053110b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f052c15c0d0 0x7f052c0cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52378/0 (socket says 172.21.15.136:52378) 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053110b640 1 -- 172.21.15.136:0/2282677938 learned_addr learned my addr 172.21.15.136:0/2282677938 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831578724 0 0) 0x7f052c15c6f0 con 0x7f052c15c0d0 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f053090a640 1 --2- 172.21.15.136:0/2282677938 >> v2:172.21.15.157:3300/0 conn(0x7f052c15ff60 0x7f052c0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.514 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.523+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0518003530 con 0x7f052c15c0d0 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2519915255 0 0) 0x7f0518003530 con 0x7f052c15c0d0 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 >> v2:172.21.15.157:3300/0 conn(0x7f052c15ff60 msgr2=0x7f052c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 --2- 172.21.15.136:0/2282677938 >> v2:172.21.15.157:3300/0 conn(0x7f052c15ff60 0x7f052c0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 >> v1:172.21.15.157:6789/0 conn(0x7f052c160370 legacy=0x7f052c166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052c15cc30 con 0x7f052c15c0d0 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f051c001680 con 0x7f052c15c0d0 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 >> v1:172.21.15.136:6789/0 conn(0x7f052c15c0d0 legacy=0x7f052c0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05180039c0 0x7f0518003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 0x7f0518004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05180051d0 0x7f051800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f052c0c6310 con 0x7f0518004810 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f051800c170 con 0x7f05180051d0 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f051800c370 con 0x7f05180039c0 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 0x7f0518004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053110b640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05180039c0 0x7f0518003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053190c640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05180051d0 0x7f051800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 -- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05180039c0 msgr2=0x7f0518003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05180039c0 0x7f0518003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 -- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05180051d0 msgr2=0x7f051800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05180051d0 0x7f051800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 -- 172.21.15.136:0/2282677938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0518003530 con 0x7f0518004810 2024-04-25T11:30:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053190c640 1 -- 172.21.15.136:0/2282677938 reap_dead start 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f053090a640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 0x7f0518004c20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0528009ae0 tx=0x7f052801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ee9bcfdab4e3b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280b2070 con 0x7f0518004810 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05280a5660 con 0x7f0518004810 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.527+0000 7f0523fff640 1 -- 172.21.15.136:0/2282677938 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280096b0 con 0x7f0518004810 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 msgr2=0x7f0518004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 0x7f0518004c20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0528009ae0 tx=0x7f052801ca70 comp rx=0 tx=0).stop 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- 172.21.15.136:0/2282677938 shutdown_connections 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 --2- 172.21.15.136:0/2282677938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0518004810 0x7f0518004c20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- 172.21.15.136:0/2282677938 >> 172.21.15.136:0/2282677938 conn(0x7f052c06f9d0 msgr2=0x7f052c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- 172.21.15.136:0/2282677938 shutdown_connections 2024-04-25T11:30:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- 172.21.15.136:0/2282677938 wait complete. 2024-04-25T11:30:31.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 Processor -- start 2024-04-25T11:30:31.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- start start 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0518004810 0x7f052c1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f052c15d950 0x7f052c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f052c15cf10 con 0x7f052c15d290 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f052c15fa40 con 0x7f0518004810 2024-04-25T11:30:31.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053336f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f052c0c9170 con 0x7f052c15d950 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44884/0 (socket says 172.21.15.136:44884) 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 -- 172.21.15.136:0/149308705 learned_addr learned my addr 172.21.15.136:0/149308705 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053110b640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0518004810 0x7f052c1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053190c640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f052c15d950 0x7f052c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 -- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f052c15d950 msgr2=0x7f052c15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f052c15d950 0x7f052c15b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 -- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0518004810 msgr2=0x7f052c1f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0518004810 0x7f052c1f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.531+0000 7f053090a640 1 -- 172.21.15.136:0/149308705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052c1f63e0 con 0x7f052c15d290 2024-04-25T11:30:31.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053090a640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f05280b0040 tx=0x7f0528019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280b2040 con 0x7f052c15d290 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0528004020 con 0x7f052c15d290 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280096b0 con 0x7f052c15d290 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f052c153c40 con 0x7f052c15d290 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f052c1f2cd0 con 0x7f052c15d290 2024-04-25T11:30:31.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f052c06a350 con 0x7f052c15d290 2024-04-25T11:30:31.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f05280bf060 con 0x7f052c15d290 2024-04-25T11:30:31.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 0x7f0510071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f0528129210 con 0x7f052c15d290 2024-04-25T11:30:31.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053110b640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 0x7f0510071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.535+0000 7f053110b640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 0x7f0510071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c10 tx=0x7f051c000ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.539+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0528016780 con 0x7f052c15d290 2024-04-25T11:30:31.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.783+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f052c0cded0 con 0x7f052c15d290 2024-04-25T11:30:31.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.783+0000 7f0521ffb640 1 -- 172.21.15.136:0/149308705 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6099 (secure 0 0 0) 0x7f05280f5630 con 0x7f052c15d290 2024-04-25T11:30:31.772 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:31.773 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4297,4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:30:31.773 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 42 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 msgr2=0x7f0510071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 0x7f0510071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f051c000c10 tx=0x7f051c000ee0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 msgr2=0x7f052c155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f05280b0040 tx=0x7f0528019810 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 shutdown_connections 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f051006f400 0x7f0510071910 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f052c15d950 0x7f052c15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f052c15d290 0x7f052c155520 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 --2- 172.21.15.136:0/149308705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0518004810 0x7f052c1f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 >> 172.21.15.136:0/149308705 conn(0x7f052c06f9d0 msgr2=0x7f052c152980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 shutdown_connections 2024-04-25T11:30:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.787+0000 7f053336f640 1 -- 172.21.15.136:0/149308705 wait complete. 2024-04-25T11:30:31.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=4269 2024-04-25T11:30:31.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2024-04-25T11:30:31.793 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-25T11:30:31.793 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T11:30:31.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T11:30:31.796 INFO:tasks.ceph.mds.b:Started 2024-04-25T11:30:31.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:31.892 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:30:31.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 Processor -- start 2024-04-25T11:30:31.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 -- start start 2024-04-25T11:30:31.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d900c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d900c9af0 con 0x7f0d900c96e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d900cb220 con 0x7f0d900c79e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d949b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9006cc10 con 0x7f0d9015aae0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0d900c96e0 0x7f0d901540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52390/0 (socket says 172.21.15.136:52390) 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8e59c640 1 -- 172.21.15.136:0/3008193030 learned_addr learned my addr 172.21.15.136:0/3008193030 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d900c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533569532 0 0) 0x7f0d900c9af0 con 0x7f0d900c96e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d78003530 con 0x7f0d900c96e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713279156 0 0) 0x7f0d900cb220 con 0x7f0d900c79e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d900c9af0 con 0x7f0d900c79e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1789301625 0 0) 0x7f0d78003530 con 0x7f0d900c96e0 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 msgr2=0x7f0d900c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d900c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 >> v1:172.21.15.157:6789/0 conn(0x7f0d900c79e0 legacy=0x7f0d9015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d900cb500 con 0x7f0d900c96e0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f0d800016a0 con 0x7f0d900c96e0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 >> v1:172.21.15.136:6789/0 conn(0x7f0d900c96e0 legacy=0x7f0d901540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d78003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d780040e0 0x7f0d78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d78004d50 0x7f0d78005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9006cc10 con 0x7f0d78004d50 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7800bb30 con 0x7f0d780040e0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7800bd30 con 0x7f0d9015aae0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d78004d50 0x7f0d78005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8e59c640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d78003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.947+0000 7f0d8dd9b640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d780040e0 0x7f0d78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 -- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 msgr2=0x7f0d78003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d9015aae0 0x7f0d78003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 -- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d780040e0 msgr2=0x7f0d78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:31.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d780040e0 0x7f0d78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 -- 172.21.15.136:0/3008193030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d78003530 con 0x7f0d78004d50 2024-04-25T11:30:31.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d78004d50 0x7f0d78005160 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009f90 tx=0x7f0d8801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e8ee75735d71ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880b2070 con 0x7f0d78004d50 2024-04-25T11:30:31.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d880a5c70 con 0x7f0d78004d50 2024-04-25T11:30:31.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8d59a640 1 -- 172.21.15.136:0/3008193030 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d88009c10 con 0x7f0d78004d50 2024-04-25T11:30:31.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 -- 172.21.15.136:0/3008193030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d78004d50 msgr2=0x7f0d78005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 --2- 172.21.15.136:0/3008193030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d78004d50 0x7f0d78005160 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009f90 tx=0x7f0d8801cc60 comp rx=0 tx=0).stop 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d8ed9d640 1 -- 172.21.15.136:0/3008193030 reap_dead start 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 -- 172.21.15.136:0/3008193030 shutdown_connections 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 -- 172.21.15.136:0/3008193030 >> 172.21.15.136:0/3008193030 conn(0x7f0d9006f9d0 msgr2=0x7f0d9006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 -- 172.21.15.136:0/3008193030 shutdown_connections 2024-04-25T11:30:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.951+0000 7f0d949b9640 1 -- 172.21.15.136:0/3008193030 wait complete. 2024-04-25T11:30:31.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 Processor -- start 2024-04-25T11:30:31.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 -- start start 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d90157300 0x7f0d90141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d90141d60 0x7f0d901f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d900ced40 con 0x7f0d90159370 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0d900cf0f0 con 0x7f0d90157300 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d949b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d901511f0 con 0x7f0d90141d60 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44902/0 (socket says 172.21.15.136:44902) 2024-04-25T11:30:31.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d8e59c640 1 -- 172.21.15.136:0/840068313 learned_addr learned my addr 172.21.15.136:0/840068313 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d8ed9d640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d90141d60 0x7f0d901f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.955+0000 7f0d8dd9b640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d90157300 0x7f0d90141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 -- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d90141d60 msgr2=0x7f0d901f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d90141d60 0x7f0d901f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 -- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d90157300 msgr2=0x7f0d90141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d90157300 0x7f0d90141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 -- 172.21.15.136:0/840068313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d900cc500 con 0x7f0d90159370 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8e59c640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0d80012470 tx=0x7f0d80012870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8001c070 con 0x7f0d90159370 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d80014810 con 0x7f0d90159370 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d90150ba0 con 0x7f0d90159370 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d80014c00 con 0x7f0d90159370 2024-04-25T11:30:31.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d900cc0d0 con 0x7f0d90159370 2024-04-25T11:30:31.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d90150990 con 0x7f0d90159370 2024-04-25T11:30:31.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0d8001f070 con 0x7f0d90159370 2024-04-25T11:30:31.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 0x7f0d70071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:31.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f0d80091fe0 con 0x7f0d90159370 2024-04-25T11:30:31.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8dd9b640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 0x7f0d70071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:31.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.959+0000 7f0d8dd9b640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 0x7f0d70071940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d900cafb0 tx=0x7f0d84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:31.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:31.967+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d8005e440 con 0x7f0d90159370 2024-04-25T11:30:32.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.219+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d9006a350 con 0x7f0d90159370 2024-04-25T11:30:32.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.219+0000 7f0d7effd640 1 -- 172.21.15.136:0/840068313 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f0d80062740 con 0x7f0d90159370 2024-04-25T11:30:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:32.208 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:32.208 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:32.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 msgr2=0x7f0d70071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 0x7f0d70071940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d900cafb0 tx=0x7f0d84009040 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 msgr2=0x7f0d90149d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0d80012470 tx=0x7f0d80012870 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 shutdown_connections 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d7006f430 0x7f0d70071940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d90141d60 0x7f0d901f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d90157300 0x7f0d90141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 --2- 172.21.15.136:0/840068313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d90159370 0x7f0d90149d30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 >> 172.21.15.136:0/840068313 conn(0x7f0d9006f9d0 msgr2=0x7f0d9006db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 shutdown_connections 2024-04-25T11:30:32.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.223+0000 7f0d949b9640 1 -- 172.21.15.136:0/840068313 wait complete. 2024-04-25T11:30:32.229 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-25T11:30:32.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:32.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 Processor -- start 2024-04-25T11:30:32.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 -- start start 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0d600b5060 0x7f0d600c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d600ca290 0x7f0d600ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d6015d750 con 0x7f0d6015a430 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6015da30 con 0x7f0d600ca290 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d67d1f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0d6015dd10 con 0x7f0d600b5060 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d600ca290 0x7f0d600ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d600ca290 0x7f0d600ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44910/0 (socket says 172.21.15.136:44910) 2024-04-25T11:30:32.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 -- 172.21.15.136:0/2090358287 learned_addr learned my addr 172.21.15.136:0/2090358287 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:32.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d65abb640 1 --2- 172.21.15.136:0/2090358287 >> v2:172.21.15.157:3300/0 conn(0x7f0d600b5060 0x7f0d600c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 -- 172.21.15.136:0/2090358287 >> v2:172.21.15.157:3300/0 conn(0x7f0d600b5060 msgr2=0x7f0d600c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:32.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 --2- 172.21.15.136:0/2090358287 >> v2:172.21.15.157:3300/0 conn(0x7f0d600b5060 0x7f0d600c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 -- 172.21.15.136:0/2090358287 >> v1:172.21.15.157:6789/0 conn(0x7f0d6015a430 legacy=0x7f0d6015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.383+0000 7f0d652ba640 1 -- 172.21.15.136:0/2090358287 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6015dff0 con 0x7f0d600ca290 2024-04-25T11:30:32.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d652ba640 1 --2- 172.21.15.136:0/2090358287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600ca290 0x7f0d600ca6a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0d60068ff0 tx=0x7f0d5001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccf271b54e4bfa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d64ab9640 1 -- 172.21.15.136:0/2090358287 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500b5070 con 0x7f0d600ca290 2024-04-25T11:30:32.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d64ab9640 1 -- 172.21.15.136:0/2090358287 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d500a8a10 con 0x7f0d600ca290 2024-04-25T11:30:32.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d64ab9640 1 -- 172.21.15.136:0/2090358287 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500a8de0 con 0x7f0d600ca290 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 -- 172.21.15.136:0/2090358287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600ca290 msgr2=0x7f0d600ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/2090358287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600ca290 0x7f0d600ca6a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0d60068ff0 tx=0x7f0d5001ed10 comp rx=0 tx=0).stop 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 -- 172.21.15.136:0/2090358287 shutdown_connections 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/2090358287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600ca290 0x7f0d600ca6a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/2090358287 >> v2:172.21.15.157:3300/0 conn(0x7f0d600b5060 0x7f0d600c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 -- 172.21.15.136:0/2090358287 >> 172.21.15.136:0/2090358287 conn(0x7f0d60070f70 msgr2=0x7f0d60068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 -- 172.21.15.136:0/2090358287 shutdown_connections 2024-04-25T11:30:32.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.387+0000 7f0d67d1f640 1 -- 172.21.15.136:0/2090358287 wait complete. 2024-04-25T11:30:32.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 Processor -- start 2024-04-25T11:30:32.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- start start 2024-04-25T11:30:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 0x7f0d600c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 0x7f0d60147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6015d3f0 con 0x7f0d600b5060 2024-04-25T11:30:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60159f10 con 0x7f0d6015a430 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0d600c4a60 con 0x7f0d600ca290 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44926/0 (socket says 172.21.15.136:44926) 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 -- 172.21.15.136:0/3371594301 learned_addr learned my addr 172.21.15.136:0/3371594301 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d652ba640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 0x7f0d600c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d662bc640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 0x7f0d60147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 -- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 msgr2=0x7f0d600c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 0x7f0d600c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 -- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 msgr2=0x7f0d60147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 0x7f0d60147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 -- 172.21.15.136:0/3371594301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6006de20 con 0x7f0d600b5060 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d662bc640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 0x7f0d60147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d652ba640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 0x7f0d600c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:32.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d65abb640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00d5f0 tx=0x7f0d5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0125f0 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5c019070 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d60153ba0 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0129d0 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.391+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6015d090 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d6015cbd0 con 0x7f0d600b5060 2024-04-25T11:30:32.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0d5c012b70 con 0x7f0d600b5060 2024-04-25T11:30:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d567fc640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 0x7f0d44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f0d5c08fcd0 con 0x7f0d600b5060 2024-04-25T11:30:32.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d652ba640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 0x7f0d44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.395+0000 7f0d652ba640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 0x7f0d44071910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0d500b2000 tx=0x7f0d500aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.403+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d6015cbd0 con 0x7f0d600b5060 2024-04-25T11:30:32.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.627+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d600b2710 con 0x7f0d600b5060 2024-04-25T11:30:32.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.627+0000 7f0d567fc640 1 -- 172.21.15.136:0/3371594301 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f0d600b2710 con 0x7f0d600b5060 2024-04-25T11:30:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:32.618 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:32.618 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:32.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 msgr2=0x7f0d44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 0x7f0d44071910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0d500b2000 tx=0x7f0d500aa070 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 msgr2=0x7f0d60152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c00d5f0 tx=0x7f0d5c00dab0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 shutdown_connections 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d6015a430 0x7f0d60147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0d4406f400 0x7f0d44071910 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0d600ca290 0x7f0d600c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 --2- 172.21.15.136:0/3371594301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d600b5060 0x7f0d60152430 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 >> 172.21.15.136:0/3371594301 conn(0x7f0d60070f70 msgr2=0x7f0d6014a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 shutdown_connections 2024-04-25T11:30:32.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.631+0000 7f0d67d1f640 1 -- 172.21.15.136:0/3371594301 wait complete. 2024-04-25T11:30:32.638 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2024-04-25T11:30:32.639 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T11:30:32.639 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:30:32.660 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T11:30:32.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:32.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 Processor -- start 2024-04-25T11:30:32.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 -- start start 2024-04-25T11:30:32.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc3300ca290 0x7fc3300ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc33015a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc33015d750 con 0x7fc3300b5060 2024-04-25T11:30:32.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc33015da30 con 0x7fc3300ca290 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc3349bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc33015dd10 con 0x7fc33015a430 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc3300b5060 0x7fc3300c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52392/0 (socket says 172.21.15.136:52392) 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32e59c640 1 -- 172.21.15.136:0/423260620 learned_addr learned my addr 172.21.15.136:0/423260620 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32ed9d640 1 --2- 172.21.15.136:0/423260620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc33015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32dd9b640 1 --2- 172.21.15.136:0/423260620 >> v2:172.21.15.157:3300/0 conn(0x7fc3300ca290 0x7fc3300ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32d59a640 1 -- 172.21.15.136:0/423260620 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979431639 0 0) 0x7fc33015d750 con 0x7fc3300b5060 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32d59a640 1 -- 172.21.15.136:0/423260620 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc318003530 con 0x7fc3300b5060 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32ed9d640 1 -- 172.21.15.136:0/423260620 >> v2:172.21.15.157:3300/0 conn(0x7fc3300ca290 msgr2=0x7fc3300ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32ed9d640 1 --2- 172.21.15.136:0/423260620 >> v2:172.21.15.157:3300/0 conn(0x7fc3300ca290 0x7fc3300ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32ed9d640 1 -- 172.21.15.136:0/423260620 >> v1:172.21.15.136:6789/0 conn(0x7fc3300b5060 legacy=0x7fc3300c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.823+0000 7fc32ed9d640 1 -- 172.21.15.136:0/423260620 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33015dff0 con 0x7fc33015a430 2024-04-25T11:30:32.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc32ed9d640 1 --2- 172.21.15.136:0/423260620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc33015a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc328009d90 tx=0x7fc32801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47da783490aaecfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc32d59a640 1 -- 172.21.15.136:0/423260620 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280b4070 con 0x7fc33015a430 2024-04-25T11:30:32.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc32d59a640 1 -- 172.21.15.136:0/423260620 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3280af600 con 0x7fc33015a430 2024-04-25T11:30:32.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc32d59a640 1 -- 172.21.15.136:0/423260620 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280af9b0 con 0x7fc33015a430 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 -- 172.21.15.136:0/423260620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 msgr2=0x7fc33015a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 --2- 172.21.15.136:0/423260620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc33015a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc328009d90 tx=0x7fc32801f150 comp rx=0 tx=0).stop 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 -- 172.21.15.136:0/423260620 shutdown_connections 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 --2- 172.21.15.136:0/423260620 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc33015a840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 --2- 172.21.15.136:0/423260620 >> v2:172.21.15.157:3300/0 conn(0x7fc3300ca290 0x7fc3300ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 -- 172.21.15.136:0/423260620 >> 172.21.15.136:0/423260620 conn(0x7fc330070f70 msgr2=0x7fc330068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 -- 172.21.15.136:0/423260620 shutdown_connections 2024-04-25T11:30:32.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.827+0000 7fc3349bf640 1 -- 172.21.15.136:0/423260620 wait complete. 2024-04-25T11:30:32.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 Processor -- start 2024-04-25T11:30:32.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- start start 2024-04-25T11:30:32.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3300ca290 0x7fc330151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc3301520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc33015ad80 con 0x7fc3300b5060 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc330159f10 con 0x7fc3300ca290 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc33015d3f0 con 0x7fc33015a430 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44938/0 (socket says 172.21.15.136:44938) 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 -- 172.21.15.136:0/1366570296 learned_addr learned my addr 172.21.15.136:0/1366570296 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32ed9d640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc3301520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32dd9b640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3300ca290 0x7fc330151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 -- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 msgr2=0x7fc3301520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc3301520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 -- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3300ca290 msgr2=0x7fc330151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3300ca290 0x7fc330151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 -- 172.21.15.136:0/1366570296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3300b3a50 con 0x7fc3300b5060 2024-04-25T11:30:32.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32ed9d640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc3301520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:30:32.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc32e59c640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c3f0 tx=0x7fc32000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320017070 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc320005cf0 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3200072f0 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3300b3cf0 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.831+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3301f2770 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33006a350 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc32001a070 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc31effd640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 0x7fc310071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7fc32008ebd0 con 0x7fc3300b5060 2024-04-25T11:30:32.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc32dd9b640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 0x7fc310071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:32.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.835+0000 7fc32dd9b640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 0x7fc310071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc3300c9ad0 tx=0x7fc324008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:32.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:32.839+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc32005af10 con 0x7fc3300b5060 2024-04-25T11:30:33.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.051+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3301577f0 con 0x7fc3300b5060 2024-04-25T11:30:33.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.051+0000 7fc31effd640 1 -- 172.21.15.136:0/1366570296 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7fc32005f210 con 0x7fc3300b5060 2024-04-25T11:30:33.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:33.042 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:33.042 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:33.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 msgr2=0x7fc310071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 0x7fc310071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc3300c9ad0 tx=0x7fc324008040 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 msgr2=0x7fc3300ce9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c3f0 tx=0x7fc32000c8b0 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 shutdown_connections 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc31006f400 0x7fc310071910 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc33015a430 0x7fc3301520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3300ca290 0x7fc330151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 --2- 172.21.15.136:0/1366570296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3300b5060 0x7fc3300ce9a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 >> 172.21.15.136:0/1366570296 conn(0x7fc330070f70 msgr2=0x7fc3300c5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 shutdown_connections 2024-04-25T11:30:33.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:33.055+0000 7fc3349bf640 1 -- 172.21.15.136:0/1366570296 wait complete. 2024-04-25T11:30:33.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:33.511 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:30:33.522+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:30:35.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:35.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.231+0000 7f38b0eac640 1 Processor -- start 2024-04-25T11:30:35.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.231+0000 7f38b0eac640 1 -- start start 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38b0eac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38b0eac640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38ac0c4340 0x7f38ac073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38b0eac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38ac0c3750 con 0x7f38ac073200 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38b0eac640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f38ac06dc00 con 0x7f38ac0c4340 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38b0eac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac14fc50 con 0x7f38ac14f840 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38ac0c4340 0x7f38ac073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38ac0c4340 0x7f38ac073610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58168/0 (socket says 172.21.15.136:58168) 2024-04-25T11:30:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 -- 172.21.15.136:0/4031003416 learned_addr learned my addr 172.21.15.136:0/4031003416 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aa59c640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 -- 172.21.15.136:0/4031003416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 msgr2=0x7f38ac074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 -- 172.21.15.136:0/4031003416 >> v1:172.21.15.157:6789/0 conn(0x7f38ac073200 legacy=0x7f38ac0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 -- 172.21.15.136:0/4031003416 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac0c3190 con 0x7f38ac0c4340 2024-04-25T11:30:35.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38aad9d640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac073610 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009e50 tx=0x7f38a401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed031f319be11e0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:35.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38a959a640 1 -- 172.21.15.136:0/4031003416 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a40a5670 con 0x7f38ac0c4340 2024-04-25T11:30:35.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38a959a640 1 -- 172.21.15.136:0/4031003416 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a40a5810 con 0x7f38ac0c4340 2024-04-25T11:30:35.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.235+0000 7f38a959a640 1 -- 172.21.15.136:0/4031003416 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a40a5be0 con 0x7f38ac0c4340 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- 172.21.15.136:0/4031003416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 msgr2=0x7f38ac073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac073610 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009e50 tx=0x7f38a401c9f0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- 172.21.15.136:0/4031003416 shutdown_connections 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac073610 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- 172.21.15.136:0/4031003416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- 172.21.15.136:0/4031003416 >> 172.21.15.136:0/4031003416 conn(0x7f38ac0ca5a0 msgr2=0x7f38ac0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- 172.21.15.136:0/4031003416 shutdown_connections 2024-04-25T11:30:35.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- 172.21.15.136:0/4031003416 wait complete. 2024-04-25T11:30:35.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 Processor -- start 2024-04-25T11:30:35.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- start start 2024-04-25T11:30:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38ac073200 0x7f38ac15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1597f0 con 0x7f38ac0c4340 2024-04-25T11:30:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac153380 con 0x7f38ac073200 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38b0eac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac159970 con 0x7f38ac14f840 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38a9d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38a9d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58176/0 (socket says 172.21.15.136:58176) 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.239+0000 7f38a9d9b640 1 -- 172.21.15.136:0/2602678177 learned_addr learned my addr 172.21.15.136:0/2602678177 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38aa59c640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38ac073200 0x7f38ac15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38aad9d640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 -- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 msgr2=0x7f38ac147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 -- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38ac073200 msgr2=0x7f38ac15b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38ac073200 0x7f38ac15b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 -- 172.21.15.136:0/2602678177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac149410 con 0x7f38ac0c4340 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38a9d9b640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f38a000b410 tx=0x7f38a000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0015030 con 0x7f38ac0c4340 2024-04-25T11:30:35.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a0004020 con 0x7f38ac0c4340 2024-04-25T11:30:35.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac15a580 con 0x7f38ac0c4340 2024-04-25T11:30:35.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0004410 con 0x7f38ac0c4340 2024-04-25T11:30:35.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac149130 con 0x7f38ac0c4340 2024-04-25T11:30:35.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac06a350 con 0x7f38ac0c4340 2024-04-25T11:30:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f38a0018070 con 0x7f38ac0c4340 2024-04-25T11:30:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 0x7f388c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f38a008f180 con 0x7f38ac0c4340 2024-04-25T11:30:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.243+0000 7f38aa59c640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 0x7f388c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.247+0000 7f38aa59c640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 0x7f388c071910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f389c006fd0 tx=0x7f389c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:35.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.251+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f38a005b550 con 0x7f38ac0c4340 2024-04-25T11:30:35.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.475+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38ac0c2ed0 con 0x7f38ac0c4340 2024-04-25T11:30:35.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f389affd640 1 -- 172.21.15.136:0/2602678177 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f38a005f850 con 0x7f38ac0c4340 2024-04-25T11:30:35.467 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:35.467 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:35.467 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:35.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 msgr2=0x7f388c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 0x7f388c071910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f389c006fd0 tx=0x7f389c008040 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 msgr2=0x7f38ac1f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f38a000b410 tx=0x7f38a000b8d0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 shutdown_connections 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f388c06f400 0x7f388c071910 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38ac14f840 0x7f38ac147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38ac0c4340 0x7f38ac1f6760 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 --2- 172.21.15.136:0/2602678177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38ac073200 0x7f38ac15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 >> 172.21.15.136:0/2602678177 conn(0x7f38ac0ca5a0 msgr2=0x7f38ac14d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 shutdown_connections 2024-04-25T11:30:35.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:35.479+0000 7f38b0eac640 1 -- 172.21.15.136:0/2602678177 wait complete. 2024-04-25T11:30:35.487 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:37.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:37.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 Processor -- start 2024-04-25T11:30:37.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 -- start start 2024-04-25T11:30:37.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b7c0c4340 0x7f1b7c164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7c0c3750 con 0x7f1b7c074980 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7c0c3190 con 0x7f1b7c14f840 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b82f54640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1b7c06dc00 con 0x7f1b7c0c4340 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b73fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1b7c074980 0x7f1b7c0c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48556/0 (socket says 172.21.15.136:48556) 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b73fff640 1 -- 172.21.15.136:0/1190724463 learned_addr learned my addr 172.21.15.136:0/1190724463 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b814f1640 1 --2- 172.21.15.136:0/1190724463 >> v2:172.21.15.136:3300/0 conn(0x7f1b7c0c4340 0x7f1b7c164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:37.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b814f1640 1 -- 172.21.15.136:0/1190724463 >> v1:172.21.15.157:6789/0 conn(0x7f1b7c14f840 legacy=0x7f1b7c073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:37.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b814f1640 1 -- 172.21.15.136:0/1190724463 >> v1:172.21.15.136:6789/0 conn(0x7f1b7c074980 legacy=0x7f1b7c0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.651+0000 7f1b814f1640 1 -- 172.21.15.136:0/1190724463 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c0c3470 con 0x7f1b7c0c4340 2024-04-25T11:30:37.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b814f1640 1 --2- 172.21.15.136:0/1190724463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c164700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1b740097f0 tx=0x7f1b7401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d3b0606e746e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:37.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b737fe640 1 -- 172.21.15.136:0/1190724463 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740b0070 con 0x7f1b7c0c4340 2024-04-25T11:30:37.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b737fe640 1 -- 172.21.15.136:0/1190724463 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b740ab600 con 0x7f1b7c0c4340 2024-04-25T11:30:37.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b737fe640 1 -- 172.21.15.136:0/1190724463 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740ab9d0 con 0x7f1b7c0c4340 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 -- 172.21.15.136:0/1190724463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c0c4340 msgr2=0x7f1b7c164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 --2- 172.21.15.136:0/1190724463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c164700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1b740097f0 tx=0x7f1b7401c460 comp rx=0 tx=0).stop 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 -- 172.21.15.136:0/1190724463 shutdown_connections 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 --2- 172.21.15.136:0/1190724463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c164700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 -- 172.21.15.136:0/1190724463 >> 172.21.15.136:0/1190724463 conn(0x7f1b7c0ca5a0 msgr2=0x7f1b7c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 -- 172.21.15.136:0/1190724463 shutdown_connections 2024-04-25T11:30:37.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.655+0000 7f1b82f54640 1 -- 172.21.15.136:0/1190724463 wait complete. 2024-04-25T11:30:37.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 Processor -- start 2024-04-25T11:30:37.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- start start 2024-04-25T11:30:37.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c1f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b7c14f840 0x7f1b7c155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c1597f0 con 0x7f1b7c074980 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c153380 con 0x7f1b7c0c4340 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7c159970 con 0x7f1b7c14f840 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58202/0 (socket says 172.21.15.136:58202) 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 -- 172.21.15.136:0/379123403 learned_addr learned my addr 172.21.15.136:0/379123403 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b73fff640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c1f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b814f1640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b7c14f840 0x7f1b7c155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:37.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 -- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b7c14f840 msgr2=0x7f1b7c155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b7c14f840 0x7f1b7c155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 -- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 msgr2=0x7f1b7c1f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c1f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 -- 172.21.15.136:0/379123403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c1f3c50 con 0x7f1b7c074980 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b73fff640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c1f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b80cf0640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800ba80 tx=0x7f1b7800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800cc00 con 0x7f1b7c074980 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7800cda0 con 0x7f1b7c074980 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7c1f6c70 con 0x7f1b7c074980 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78003bc0 con 0x7f1b7c074980 2024-04-25T11:30:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.659+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7c1f7230 con 0x7f1b7c074980 2024-04-25T11:30:37.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7c1f34f0 con 0x7f1b7c074980 2024-04-25T11:30:37.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1b78006d10 con 0x7f1b7c074980 2024-04-25T11:30:37.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b717fa640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 0x7f1b68071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:37.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f1b7808f3f0 con 0x7f1b7c074980 2024-04-25T11:30:37.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b73fff640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 0x7f1b68071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:37.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.663+0000 7f1b73fff640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 0x7f1b68071910 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0cc2d0 tx=0x7f1b6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:37.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.667+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1b7805bea0 con 0x7f1b7c074980 2024-04-25T11:30:37.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.883+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b7c06a350 con 0x7f1b7c074980 2024-04-25T11:30:37.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.883+0000 7f1b717fa640 1 -- 172.21.15.136:0/379123403 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f1b7805b7a0 con 0x7f1b7c074980 2024-04-25T11:30:37.874 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:37.874 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:37.874 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 msgr2=0x7f1b68071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 0x7f1b68071910 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0cc2d0 tx=0x7f1b6c009040 comp rx=0 tx=0).stop 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 msgr2=0x7f1b7c14ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800ba80 tx=0x7f1b7800bf40 comp rx=0 tx=0).stop 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 shutdown_connections 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1b6806f400 0x7f1b68071910 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b7c14f840 0x7f1b7c155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b7c0c4340 0x7f1b7c1f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 --2- 172.21.15.136:0/379123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b7c074980 0x7f1b7c14ec10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 >> 172.21.15.136:0/379123403 conn(0x7f1b7c0ca5a0 msgr2=0x7f1b7c14e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 shutdown_connections 2024-04-25T11:30:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:37.887+0000 7f1b82f54640 1 -- 172.21.15.136:0/379123403 wait complete. 2024-04-25T11:30:37.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:39.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:40.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.075+0000 7f86eace2640 1 Processor -- start 2024-04-25T11:30:40.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.075+0000 7f86eace2640 1 -- start start 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86eace2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f86e40b5810 0x7f86e40b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86eace2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 0x7f86e406d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86eace2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e40b4480 con 0x7f86e415c480 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86eace2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f86e40b4a90 con 0x7f86e40b5810 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86eace2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f86e415c990 con 0x7f86e415c070 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f86e40b5810 0x7f86e40b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e927f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f86e415c480 0x7f86e415f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:48562/0 (socket says 172.21.15.136:48562) 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e927f640 1 -- 172.21.15.136:0/1655621512 learned_addr learned my addr 172.21.15.136:0/1655621512 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e3fff640 1 --2- 172.21.15.136:0/1655621512 >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 0x7f86e406d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 -- 172.21.15.136:0/1655621512 >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 msgr2=0x7f86e406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 --2- 172.21.15.136:0/1655621512 >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 0x7f86e406d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 -- 172.21.15.136:0/1655621512 >> v1:172.21.15.136:6789/0 conn(0x7f86e415c480 legacy=0x7f86e415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 -- 172.21.15.136:0/1655621512 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e415cc70 con 0x7f86e40b5810 2024-04-25T11:30:40.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e3fff640 1 --2- 172.21.15.136:0/1655621512 >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 0x7f86e406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:40.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e8a7e640 1 --2- 172.21.15.136:0/1655621512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e40b5810 0x7f86e40b4f20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f86e4069490 tx=0x7f86d801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cf1d60c7e89325 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:40.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.079+0000 7f86e37fe640 1 -- 172.21.15.136:0/1655621512 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d80b5070 con 0x7f86e40b5810 2024-04-25T11:30:40.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86e37fe640 1 -- 172.21.15.136:0/1655621512 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d80a8970 con 0x7f86e40b5810 2024-04-25T11:30:40.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86e37fe640 1 -- 172.21.15.136:0/1655621512 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d80a8d40 con 0x7f86e40b5810 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- 172.21.15.136:0/1655621512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e40b5810 msgr2=0x7f86e40b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 --2- 172.21.15.136:0/1655621512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e40b5810 0x7f86e40b4f20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f86e4069490 tx=0x7f86d801ec70 comp rx=0 tx=0).stop 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- 172.21.15.136:0/1655621512 shutdown_connections 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 --2- 172.21.15.136:0/1655621512 >> v2:172.21.15.157:3300/0 conn(0x7f86e415c070 0x7f86e406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 --2- 172.21.15.136:0/1655621512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e40b5810 0x7f86e40b4f20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- 172.21.15.136:0/1655621512 >> 172.21.15.136:0/1655621512 conn(0x7f86e40743a0 msgr2=0x7f86e40d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- 172.21.15.136:0/1655621512 shutdown_connections 2024-04-25T11:30:40.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- 172.21.15.136:0/1655621512 wait complete. 2024-04-25T11:30:40.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 Processor -- start 2024-04-25T11:30:40.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.083+0000 7f86eace2640 1 -- start start 2024-04-25T11:30:40.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86e40b5810 0x7f86e41f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86e415c070 0x7f86e41e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f86e406e030 con 0x7f86e415c480 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f86e415cf50 con 0x7f86e415c070 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f86e40b5460 con 0x7f86e40b5810 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58218/0 (socket says 172.21.15.136:58218) 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 -- 172.21.15.136:0/970093202 learned_addr learned my addr 172.21.15.136:0/970093202 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e3fff640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86e415c070 0x7f86e41e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e8a7e640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86e40b5810 0x7f86e41f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 -- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86e40b5810 msgr2=0x7f86e41f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86e40b5810 0x7f86e41f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 -- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86e415c070 msgr2=0x7f86e41e75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86e415c070 0x7f86e41e75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 -- 172.21.15.136:0/970093202 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e41f2e00 con 0x7f86e415c480 2024-04-25T11:30:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e927f640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00caa0 tx=0x7f86dc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:40.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc017070 con 0x7f86e415c480 2024-04-25T11:30:40.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86dc00aba0 con 0x7f86e415c480 2024-04-25T11:30:40.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e415ff20 con 0x7f86e415c480 2024-04-25T11:30:40.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc012400 con 0x7f86e415c480 2024-04-25T11:30:40.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e41f3010 con 0x7f86e415c480 2024-04-25T11:30:40.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.087+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e406a350 con 0x7f86e415c480 2024-04-25T11:30:40.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.091+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f86dc01a070 con 0x7f86e415c480 2024-04-25T11:30:40.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.091+0000 7f86e17fa640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 0x7f86c8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:40.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.091+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f86dc08f3f0 con 0x7f86e415c480 2024-04-25T11:30:40.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.091+0000 7f86e8a7e640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 0x7f86c8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:40.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.091+0000 7f86e8a7e640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 0x7f86c8071910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86d80b2000 tx=0x7f86d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:40.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.099+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f86dc05bea0 con 0x7f86e415c480 2024-04-25T11:30:40.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.383+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86e41f2200 con 0x7f86e415c480 2024-04-25T11:30:40.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86e17fa640 1 -- 172.21.15.136:0/970093202 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f86dc05b7a0 con 0x7f86e415c480 2024-04-25T11:30:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:40.375 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:40.375 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:40.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 msgr2=0x7f86c8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 0x7f86c8071910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f86d80b2000 tx=0x7f86d8008040 comp rx=0 tx=0).stop 2024-04-25T11:30:40.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 msgr2=0x7f86e4153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:40.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00caa0 tx=0x7f86dc00cf60 comp rx=0 tx=0).stop 2024-04-25T11:30:40.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 shutdown_connections 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86e415c480 0x7f86e4153870 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86e415c070 0x7f86e41e75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f86c806f400 0x7f86c8071910 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 --2- 172.21.15.136:0/970093202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86e40b5810 0x7f86e41f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 >> 172.21.15.136:0/970093202 conn(0x7f86e40743a0 msgr2=0x7f86e4155a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 shutdown_connections 2024-04-25T11:30:40.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:40.387+0000 7f86eace2640 1 -- 172.21.15.136:0/970093202 wait complete. 2024-04-25T11:30:40.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:42.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:42.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.559+0000 7fa6da871640 1 Processor -- start 2024-04-25T11:30:42.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.559+0000 7fa6da871640 1 -- start start 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6da871640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6da871640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 0x7fa6d40b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6da871640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d40ccb80 con 0x7fa6d4156330 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6da871640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa6d41602f0 con 0x7fa6d40cdee0 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6da871640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d4156740 con 0x7fa6d40b5490 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58226/0 (socket says 172.21.15.136:58226) 2024-04-25T11:30:42.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 -- 172.21.15.136:0/430823291 learned_addr learned my addr 172.21.15.136:0/430823291 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6d8e0e640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 0x7fa6d40b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 -- 172.21.15.136:0/430823291 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 msgr2=0x7fa6d40b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 0x7fa6d40b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 -- 172.21.15.136:0/430823291 >> v1:172.21.15.136:6789/0 conn(0x7fa6d4156330 legacy=0x7fa6d415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6d37fe640 1 -- 172.21.15.136:0/430823291 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139682722 0 0) 0x7fa6d40ccb80 con 0x7fa6d4156330 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 -- 172.21.15.136:0/430823291 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d415f6e0 con 0x7fa6d40cdee0 2024-04-25T11:30:42.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6d8e0e640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 0x7fa6d40b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:42.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.563+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009870 tx=0x7fa6bc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed462bd97ed2ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:42.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6d37fe640 1 -- 172.21.15.136:0/430823291 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0b5070 con 0x7fa6d40cdee0 2024-04-25T11:30:42.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6d37fe640 1 -- 172.21.15.136:0/430823291 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6bc0a8a50 con 0x7fa6d40cdee0 2024-04-25T11:30:42.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6d37fe640 1 -- 172.21.15.136:0/430823291 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0a8e20 con 0x7fa6d40cdee0 2024-04-25T11:30:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- 172.21.15.136:0/430823291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40cdee0 msgr2=0x7fa6d40ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009870 tx=0x7fa6bc01ed50 comp rx=0 tx=0).stop 2024-04-25T11:30:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- 172.21.15.136:0/430823291 shutdown_connections 2024-04-25T11:30:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d40b5490 0x7fa6d40b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 --2- 172.21.15.136:0/430823291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40ce2f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- 172.21.15.136:0/430823291 >> 172.21.15.136:0/430823291 conn(0x7fa6d406d6c0 msgr2=0x7fa6d406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:42.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- 172.21.15.136:0/430823291 shutdown_connections 2024-04-25T11:30:42.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- 172.21.15.136:0/430823291 wait complete. 2024-04-25T11:30:42.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 Processor -- start 2024-04-25T11:30:42.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.567+0000 7fa6da871640 1 -- start start 2024-04-25T11:30:42.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 0x7fa6d41e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d415ff80 con 0x7fa6d40b5490 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d415fca0 con 0x7fa6d40cdee0 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d40b5de0 con 0x7fa6d4156330 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58236/0 (socket says 172.21.15.136:58236) 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 -- 172.21.15.136:0/3647392442 learned_addr learned my addr 172.21.15.136:0/3647392442 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d8e0e640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 0x7fa6d41e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 -- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 msgr2=0x7fa6d41e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 0x7fa6d41e79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 -- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 msgr2=0x7fa6d40c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 -- 172.21.15.136:0/3647392442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d40c95f0 con 0x7fa6d40b5490 2024-04-25T11:30:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d8e0e640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 0x7fa6d41e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d3fff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400c450 tx=0x7fa6c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4017070 con 0x7fa6d40b5490 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c4003600 con 0x7fa6d40b5490 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d40c88f0 con 0x7fa6d40b5490 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40039f0 con 0x7fa6d40b5490 2024-04-25T11:30:42.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d40c8b90 con 0x7fa6d40b5490 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d40c97c0 con 0x7fa6d40b5490 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.571+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa6c401a070 con 0x7fa6d40b5490 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.575+0000 7fa6d17fa640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 0x7fa6b4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.575+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7fa6c408eb00 con 0x7fa6d40b5490 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.575+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 0x7fa6b4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:42.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.575+0000 7fa6cbfff640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 0x7fa6b4071910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0b2000 tx=0x7fa6bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:42.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.579+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa6c405b5b0 con 0x7fa6d40b5490 2024-04-25T11:30:42.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6d406a350 con 0x7fa6d40b5490 2024-04-25T11:30:42.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6d17fa640 1 -- 172.21.15.136:0/3647392442 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7fa6c405aeb0 con 0x7fa6d40b5490 2024-04-25T11:30:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:42.820 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:42.820 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 msgr2=0x7fa6b4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 0x7fa6b4071910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc0b2000 tx=0x7fa6bc008040 comp rx=0 tx=0).stop 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 msgr2=0x7fa6d40d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400c450 tx=0x7fa6c400c910 comp rx=0 tx=0).stop 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 shutdown_connections 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b406f400 0x7fa6b4071910 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d4156330 0x7fa6d41e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6d40cdee0 0x7fa6d40c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 --2- 172.21.15.136:0/3647392442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d40b5490 0x7fa6d40d16b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 >> 172.21.15.136:0/3647392442 conn(0x7fa6d406d6c0 msgr2=0x7fa6d4154c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 shutdown_connections 2024-04-25T11:30:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:42.831+0000 7fa6da871640 1 -- 172.21.15.136:0/3647392442 wait complete. 2024-04-25T11:30:42.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:44.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:45.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 Processor -- start 2024-04-25T11:30:45.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 -- start start 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 0x7ff4a00cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff4a015ff60 0x7ff4a00ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a015c6f0 con 0x7ff4a0160370 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a00c6310 con 0x7ff4a015ff60 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a7ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a015c990 con 0x7ff4a015c0d0 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff4a015ff60 0x7ff4a00ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff4a015ff60 0x7ff4a00ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:53970/0 (socket says 172.21.15.136:53970) 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 -- 172.21.15.136:0/4280964733 learned_addr learned my addr 172.21.15.136:0/4280964733 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a5d90640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 0x7ff4a00cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 -- 172.21.15.136:0/4280964733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 msgr2=0x7ff4a00cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 0x7ff4a00cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 -- 172.21.15.136:0/4280964733 >> v1:172.21.15.157:6789/0 conn(0x7ff4a0160370 legacy=0x7ff4a0071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a558f640 1 -- 172.21.15.136:0/4280964733 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a015cc30 con 0x7ff4a015ff60 2024-04-25T11:30:45.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.031+0000 7ff4a5d90640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 0x7ff4a00cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:45.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a558f640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a00ce260 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff490009870 tx=0x7ff49001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0af4c1546b8efe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:45.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a4d8e640 1 -- 172.21.15.136:0/4280964733 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4900b5070 con 0x7ff4a015ff60 2024-04-25T11:30:45.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a4d8e640 1 -- 172.21.15.136:0/4280964733 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4900a89d0 con 0x7ff4a015ff60 2024-04-25T11:30:45.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a4d8e640 1 -- 172.21.15.136:0/4280964733 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4900a8da0 con 0x7ff4a015ff60 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/4280964733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 msgr2=0x7ff4a00ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a00ce260 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff490009870 tx=0x7ff49001ed10 comp rx=0 tx=0).stop 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/4280964733 shutdown_connections 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a00ce260 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/4280964733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a015c0d0 0x7ff4a00cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/4280964733 >> 172.21.15.136:0/4280964733 conn(0x7ff4a006f9d0 msgr2=0x7ff4a006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:45.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/4280964733 shutdown_connections 2024-04-25T11:30:45.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.035+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/4280964733 wait complete. 2024-04-25T11:30:45.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 Processor -- start 2024-04-25T11:30:45.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- start start 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a015c0d0 0x7ff4a0153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 0x7ff4a01528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a00cbc60 con 0x7ff4a015ff60 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a015fa40 con 0x7ff4a015c0d0 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a015cf10 con 0x7ff4a0160370 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:53974/0 (socket says 172.21.15.136:53974) 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 -- 172.21.15.136:0/2827753914 learned_addr learned my addr 172.21.15.136:0/2827753914 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:45.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a5d90640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a015c0d0 0x7ff4a0153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a6591640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 0x7ff4a01528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 -- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 msgr2=0x7ff4a01528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 0x7ff4a01528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 -- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a015c0d0 msgr2=0x7ff4a0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a015c0d0 0x7ff4a0153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 -- 172.21.15.136:0/2827753914 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a0153f70 con 0x7ff4a015ff60 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a6591640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 0x7ff4a01528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a558f640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff4900b3040 tx=0x7ff49001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4900b5070 con 0x7ff4a015ff60 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4900b0b70 con 0x7ff4a015ff60 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a014fef0 con 0x7ff4a015ff60 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4900bb410 con 0x7ff4a015ff60 2024-04-25T11:30:45.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.039+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a015e5c0 con 0x7ff4a015ff60 2024-04-25T11:30:45.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a006a350 con 0x7ff4a015ff60 2024-04-25T11:30:45.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff4900c2060 con 0x7ff4a015ff60 2024-04-25T11:30:45.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4967fc640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 0x7ff484071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:45.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7ff49012c940 con 0x7ff4a015ff60 2024-04-25T11:30:45.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4a5d90640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 0x7ff484071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:45.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.043+0000 7ff4a5d90640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 0x7ff484071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff49c006fd0 tx=0x7ff49c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:45.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.051+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff490018870 con 0x7ff4a015ff60 2024-04-25T11:30:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.279+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4a00ca8b0 con 0x7ff4a015ff60 2024-04-25T11:30:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.279+0000 7ff4967fc640 1 -- 172.21.15.136:0/2827753914 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7ff4900f9360 con 0x7ff4a015ff60 2024-04-25T11:30:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:45.270 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:45.270 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 msgr2=0x7ff484071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 0x7ff484071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff49c006fd0 tx=0x7ff49c008040 comp rx=0 tx=0).stop 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 msgr2=0x7ff4a01523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff4900b3040 tx=0x7ff49001bab0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 shutdown_connections 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff48406f400 0x7ff484071910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4a0160370 0x7ff4a01528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4a015ff60 0x7ff4a01523a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 --2- 172.21.15.136:0/2827753914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a015c0d0 0x7ff4a0153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:45.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 >> 172.21.15.136:0/2827753914 conn(0x7ff4a006f9d0 msgr2=0x7ff4a015b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:45.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 shutdown_connections 2024-04-25T11:30:45.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:45.283+0000 7ff4a7ff4640 1 -- 172.21.15.136:0/2827753914 wait complete. 2024-04-25T11:30:45.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:47.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.475+0000 7f0c0f41d640 1 Processor -- start 2024-04-25T11:30:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.475+0000 7f0c0f41d640 1 -- start start 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0f41d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0c0815aae0 0x7f0c080c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0f41d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c080c9af0 con 0x7f0c080c79e0 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0f41d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c080cb220 con 0x7f0c080c96e0 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0f41d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0c0806cc10 con 0x7f0c0815aae0 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0c9b8640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0c080c79e0 0x7f0c0815a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33918/0 (socket says 172.21.15.136:33918) 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0c9b8640 1 -- 172.21.15.136:0/3923874021 learned_addr learned my addr 172.21.15.136:0/3923874021 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3923874021 >> v2:172.21.15.157:3300/0 conn(0x7f0c0815aae0 0x7f0c080c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789701455 0 0) 0x7f0c080c9af0 con 0x7f0c080c79e0 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bf4003530 con 0x7f0c080c79e0 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1900405659 0 0) 0x7f0bf4003530 con 0x7f0c080c79e0 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 >> v2:172.21.15.157:3300/0 conn(0x7f0c0815aae0 msgr2=0x7f0c080c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 --2- 172.21.15.136:0/3923874021 >> v2:172.21.15.157:3300/0 conn(0x7f0c0815aae0 0x7f0c080c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 >> v1:172.21.15.157:6789/0 conn(0x7f0c080c96e0 legacy=0x7f0c081540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c080cb500 con 0x7f0c080c79e0 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f0bf8001680 con 0x7f0c080c79e0 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 >> v1:172.21.15.136:6789/0 conn(0x7f0c080c79e0 legacy=0x7f0c0815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bf40039c0 0x7f0bf4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bf4004810 0x7f0bf4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0bf400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0806cc10 con 0x7f0bf4005280 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf400c1a0 con 0x7f0bf40039c0 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf400c3a0 con 0x7f0bf4004810 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0bf400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 -- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bf4004810 msgr2=0x7f0bf4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bf4004810 0x7f0bf4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d1b9640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bf40039c0 0x7f0bf4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 -- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bf40039c0 msgr2=0x7f0bf4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bf40039c0 0x7f0bf4003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.479+0000 7f0c0d9ba640 1 -- 172.21.15.136:0/3923874021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf4003530 con 0x7f0bf4005280 2024-04-25T11:30:47.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.483+0000 7f0c0d9ba640 1 -- 172.21.15.136:0/3923874021 reap_dead start 2024-04-25T11:30:47.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.483+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0bf400b9d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009fc0 tx=0x7f0c0001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414a982a0b6ec3b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:47.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.483+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00009c10 con 0x7f0bf4005280 2024-04-25T11:30:47.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.483+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c000a5cd0 con 0x7f0bf4005280 2024-04-25T11:30:47.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.483+0000 7f0bfffff640 1 -- 172.21.15.136:0/3923874021 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00003be0 con 0x7f0bf4005280 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 msgr2=0x7f0bf400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0bf400b9d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009fc0 tx=0x7f0c0001ccc0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3923874021 shutdown_connections 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3923874021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0bf400b9d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3923874021 >> 172.21.15.136:0/3923874021 conn(0x7f0c0806f9d0 msgr2=0x7f0c0806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3923874021 shutdown_connections 2024-04-25T11:30:47.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3923874021 wait complete. 2024-04-25T11:30:47.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 Processor -- start 2024-04-25T11:30:47.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.491+0000 7f0c0f41d640 1 -- start start 2024-04-25T11:30:47.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c0815bc90 0x7f0c08155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c080cd420 0x7f0c08149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0806d5c0 con 0x7f0bf4005280 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0806d440 con 0x7f0c0815bc90 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0c080ced40 con 0x7f0c080cd420 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:53994/0 (socket says 172.21.15.136:53994) 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 -- 172.21.15.136:0/3390376390 learned_addr learned my addr 172.21.15.136:0/3390376390 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0c9b8640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c0815bc90 0x7f0c08155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d9ba640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c080cd420 0x7f0c08149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 -- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c080cd420 msgr2=0x7f0c08149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c080cd420 0x7f0c08149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 -- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c0815bc90 msgr2=0x7f0c08155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c0815bc90 0x7f0c08155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 -- 172.21.15.136:0/3390376390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c08156450 con 0x7f0bf4005280 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0d1b9640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d570 tx=0x7f0c0400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04005480 con 0x7f0bf4005280 2024-04-25T11:30:47.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0815b2e0 con 0x7f0bf4005280 2024-04-25T11:30:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c04017070 con 0x7f0bf4005280 2024-04-25T11:30:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c081559f0 con 0x7f0bf4005280 2024-04-25T11:30:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04003820 con 0x7f0bf4005280 2024-04-25T11:30:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.495+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0806a350 con 0x7f0bf4005280 2024-04-25T11:30:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.499+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0c040039c0 con 0x7f0bf4005280 2024-04-25T11:30:47.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.499+0000 7f0bfdffb640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 0x7f0bec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:47.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.499+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 7079+0+0 (secure 0 0 0) 0x7f0c0408f550 con 0x7f0bf4005280 2024-04-25T11:30:47.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.499+0000 7f0c0c9b8640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 0x7f0bec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:47.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.503+0000 7f0c0c9b8640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 0x7f0bec071910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8001870 tx=0x7f0bf80016c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:47.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.507+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0c0405bf80 con 0x7f0bf4005280 2024-04-25T11:30:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.747+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c081579a0 con 0x7f0bf4005280 2024-04-25T11:30:47.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.747+0000 7f0bfdffb640 1 -- 172.21.15.136:0/3390376390 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6926 (secure 0 0 0) 0x7f0c0405b880 con 0x7f0bf4005280 2024-04-25T11:30:47.736 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:47.736 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:30.141516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4297,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":12,"addr":"172.21.15.157:6836/694883637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":694883637},{"type":"v1","addr":"172.21.15.157:6836","nonce":694883637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303,4297]},"id":1}]} 2024-04-25T11:30:47.737 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 43 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 msgr2=0x7f0bec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 0x7f0bec071910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8001870 tx=0x7f0bf80016c0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 msgr2=0x7f0c081f64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400d570 tx=0x7f0c0400da30 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 shutdown_connections 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0bec06f400 0x7f0bec071910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c080cd420 0x7f0c08149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c0815bc90 0x7f0c08155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 --2- 172.21.15.136:0/3390376390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bf4005280 0x7f0c081f64a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 >> 172.21.15.136:0/3390376390 conn(0x7f0c0806f9d0 msgr2=0x7f0c081534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:47.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 shutdown_connections 2024-04-25T11:30:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:47.751+0000 7f0c0f41d640 1 -- 172.21.15.136:0/3390376390 wait complete. 2024-04-25T11:30:47.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:30:49.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:49.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 Processor -- start 2024-04-25T11:30:49.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 -- start start 2024-04-25T11:30:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6d815a5f0 0x7fd6d80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d80c8d40 con 0x7fd6d815aa00 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d80c8fd0 con 0x7fd6d815a5f0 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6def14640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d80c3880 con 0x7fd6d8154060 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6d815a5f0 0x7fd6d80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6d815a5f0 0x7fd6d80c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54000/0 (socket says 172.21.15.136:54000) 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 -- 172.21.15.136:0/699273931 learned_addr learned my addr 172.21.15.136:0/699273931 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 -- 172.21.15.136:0/699273931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 msgr2=0x7fd6d80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d80c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 -- 172.21.15.136:0/699273931 >> v1:172.21.15.157:6789/0 conn(0x7fd6d815aa00 legacy=0x7fd6d815d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6d7fff640 1 -- 172.21.15.136:0/699273931 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d806d9b0 con 0x7fd6d815a5f0 2024-04-25T11:30:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.935+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:49.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6d7fff640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d80c71b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd6d806db80 tx=0x7fd6c401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b2651c27525826 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:49.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6d77fe640 1 -- 172.21.15.136:0/699273931 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40b5070 con 0x7fd6d815a5f0 2024-04-25T11:30:49.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6d77fe640 1 -- 172.21.15.136:0/699273931 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c40a8a30 con 0x7fd6d815a5f0 2024-04-25T11:30:49.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6d77fe640 1 -- 172.21.15.136:0/699273931 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40a8e00 con 0x7fd6d815a5f0 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 -- 172.21.15.136:0/699273931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 msgr2=0x7fd6d80c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d80c71b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd6d806db80 tx=0x7fd6c401ed30 comp rx=0 tx=0).stop 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 -- 172.21.15.136:0/699273931 shutdown_connections 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d80c71b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 --2- 172.21.15.136:0/699273931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 -- 172.21.15.136:0/699273931 >> 172.21.15.136:0/699273931 conn(0x7fd6d80ca5a0 msgr2=0x7fd6d80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 -- 172.21.15.136:0/699273931 shutdown_connections 2024-04-25T11:30:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.939+0000 7fd6def14640 1 -- 172.21.15.136:0/699273931 wait complete. 2024-04-25T11:30:49.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 Processor -- start 2024-04-25T11:30:49.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- start start 2024-04-25T11:30:49.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d81e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 0x7fd6d81442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d806df30 con 0x7fd6d815a5f0 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d8154470 con 0x7fd6d815aa00 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d815a0d0 con 0x7fd6d8154060 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54014/0 (socket says 172.21.15.136:54014) 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 -- 172.21.15.136:0/2390091858 learned_addr learned my addr 172.21.15.136:0/2390091858 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d81e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6dd4b1640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 0x7fd6d81442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 -- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 msgr2=0x7fd6d81e5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d81e5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 -- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 msgr2=0x7fd6d81442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 0x7fd6d81442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 -- 172.21.15.136:0/2390091858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d81e5e90 con 0x7fd6d815a5f0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d81e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6dd4b1640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 0x7fd6d81442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d7fff640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c10 tx=0x7fd6c401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40b5070 con 0x7fd6d815a5f0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c40b0b40 con 0x7fd6d815a5f0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40bb410 con 0x7fd6d815a5f0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d8068870 con 0x7fd6d815a5f0 2024-04-25T11:30:49.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.943+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d80cd3d0 con 0x7fd6d815a5f0 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d806a350 con 0x7fd6d815a5f0 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6c40b9070 con 0x7fd6d815a5f0 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6d57fa640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 0x7fd6bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fd6c412c890 con 0x7fd6d815a5f0 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 0x7fd6bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:49.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.947+0000 7fd6dccb0640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 0x7fd6bc071910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00b990 tx=0x7fd6cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:49.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:49.955+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6c40c1060 con 0x7fd6d815a5f0 2024-04-25T11:30:50.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.179+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d80c85e0 con 0x7fd6d815a5f0 2024-04-25T11:30:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.179+0000 7fd6d57fa640 1 -- 172.21.15.136:0/2390091858 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6096 (secure 0 0 0) 0x7fd6c4018890 con 0x7fd6d815a5f0 2024-04-25T11:30:50.169 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:50.169 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:48.888045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:30:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 44 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 msgr2=0x7fd6bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 0x7fd6bc071910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00b990 tx=0x7fd6cc008040 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 msgr2=0x7fd6d81f38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c10 tx=0x7fd6c401bad0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 shutdown_connections 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6d815aa00 0x7fd6d81442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d815a5f0 0x7fd6d81f38f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd6bc06f400 0x7fd6bc071910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 --2- 172.21.15.136:0/2390091858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6d8154060 0x7fd6d81e5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 >> 172.21.15.136:0/2390091858 conn(0x7fd6d80ca5a0 msgr2=0x7fd6d814d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 shutdown_connections 2024-04-25T11:30:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.183+0000 7fd6def14640 1 -- 172.21.15.136:0/2390091858 wait complete. 2024-04-25T11:30:50.191 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-04-25T11:30:50.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:30:50.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:50.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.355+0000 7f5fe562a640 1 Processor -- start 2024-04-25T11:30:50.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.355+0000 7f5fe562a640 1 -- start start 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fe562a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 0x7f5fe00c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fe562a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5fe00ca290 0x7f5fe00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fe562a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fe015d750 con 0x7f5fe015a430 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fe562a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5fe015da30 con 0x7f5fe00ca290 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fe562a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5fe015dd10 con 0x7f5fe00b5060 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5fe00ca290 0x7f5fe00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fdf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5fe015a430 0x7f5fe015a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33924/0 (socket says 172.21.15.136:33924) 2024-04-25T11:30:50.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fdf7fe640 1 -- 172.21.15.136:0/2952495688 learned_addr learned my addr 172.21.15.136:0/2952495688 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fdeffd640 1 --2- 172.21.15.136:0/2952495688 >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 0x7f5fe00c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 -- 172.21.15.136:0/2952495688 >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 msgr2=0x7f5fe00c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 --2- 172.21.15.136:0/2952495688 >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 0x7f5fe00c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 -- 172.21.15.136:0/2952495688 >> v1:172.21.15.136:6789/0 conn(0x7f5fe015a430 legacy=0x7f5fe015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 -- 172.21.15.136:0/2952495688 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe015dff0 con 0x7f5fe00ca290 2024-04-25T11:30:50.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fdeffd640 1 --2- 172.21.15.136:0/2952495688 >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 0x7f5fe00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:50.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fde7fc640 1 --2- 172.21.15.136:0/2952495688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe00ca290 0x7f5fe00ca6a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0068ff0 tx=0x7f5fd401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b4d1c5f99dad92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:50.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fddffb640 1 -- 172.21.15.136:0/2952495688 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40b5070 con 0x7f5fe00ca290 2024-04-25T11:30:50.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fddffb640 1 -- 172.21.15.136:0/2952495688 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd40a8970 con 0x7f5fe00ca290 2024-04-25T11:30:50.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.359+0000 7f5fddffb640 1 -- 172.21.15.136:0/2952495688 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40a8d40 con 0x7f5fe00ca290 2024-04-25T11:30:50.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- 172.21.15.136:0/2952495688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe00ca290 msgr2=0x7f5fe00ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- 172.21.15.136:0/2952495688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe00ca290 0x7f5fe00ca6a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0068ff0 tx=0x7f5fd401ec70 comp rx=0 tx=0).stop 2024-04-25T11:30:50.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- 172.21.15.136:0/2952495688 shutdown_connections 2024-04-25T11:30:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- 172.21.15.136:0/2952495688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe00ca290 0x7f5fe00ca6a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- 172.21.15.136:0/2952495688 >> v2:172.21.15.157:3300/0 conn(0x7f5fe00b5060 0x7f5fe00c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- 172.21.15.136:0/2952495688 >> 172.21.15.136:0/2952495688 conn(0x7f5fe0070f70 msgr2=0x7f5fe0068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- 172.21.15.136:0/2952495688 shutdown_connections 2024-04-25T11:30:50.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- 172.21.15.136:0/2952495688 wait complete. 2024-04-25T11:30:50.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 Processor -- start 2024-04-25T11:30:50.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- start start 2024-04-25T11:30:50.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe00b5060 0x7f5fe01f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fe00ca290 0x7f5fe0147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe015d3f0 con 0x7f5fe015a430 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe0159f10 con 0x7f5fe00b5060 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fe562a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe00c4a60 con 0x7f5fe00ca290 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fdf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fdf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54034/0 (socket says 172.21.15.136:54034) 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fdf7fe640 1 -- 172.21.15.136:0/658552261 learned_addr learned my addr 172.21.15.136:0/658552261 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fdeffd640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe00b5060 0x7f5fe01f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.363+0000 7f5fde7fc640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fe00ca290 0x7f5fe0147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 -- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fe00ca290 msgr2=0x7f5fe0147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fe00ca290 0x7f5fe0147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 -- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe00b5060 msgr2=0x7f5fe01f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe00b5060 0x7f5fe01f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 -- 172.21.15.136:0/658552261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe01f1530 con 0x7f5fe015a430 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdf7fe640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800ca20 tx=0x7f5fd800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8017070 con 0x7f5fe015a430 2024-04-25T11:30:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd800ab70 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe015b0d0 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8012400 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe01f07f0 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe015b5f0 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5fd801a070 con 0x7f5fe015a430 2024-04-25T11:30:50.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 0x7f5fcc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:50.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f5fd808f280 con 0x7f5fe015a430 2024-04-25T11:30:50.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.367+0000 7f5fdeffd640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 0x7f5fcc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:50.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.371+0000 7f5fdeffd640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 0x7f5fcc071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006fd0 tx=0x7f5fd0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:50.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.375+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5fd805b690 con 0x7f5fe015a430 2024-04-25T11:30:50.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.603+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fe01f05e0 con 0x7f5fe015a430 2024-04-25T11:30:50.591 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.603+0000 7f5fc77fe640 1 -- 172.21.15.136:0/658552261 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6096 (secure 0 0 0) 0x7f5fd805f990 con 0x7f5fe015a430 2024-04-25T11:30:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:50.591 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:48.888045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:30:50.592 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 44 2024-04-25T11:30:50.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 msgr2=0x7f5fcc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 0x7f5fcc071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006fd0 tx=0x7f5fd0008040 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 msgr2=0x7f5fe0153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800ca20 tx=0x7f5fd800cee0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 shutdown_connections 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe015a430 0x7f5fe0153040 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fcc06f400 0x7f5fcc071910 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fe00ca290 0x7f5fe0147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 --2- 172.21.15.136:0/658552261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fe00b5060 0x7f5fe01f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 >> 172.21.15.136:0/658552261 conn(0x7f5fe0070f70 msgr2=0x7f5fe00cb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 shutdown_connections 2024-04-25T11:30:50.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:50.607+0000 7f5fe562a640 1 -- 172.21.15.136:0/658552261 wait complete. 2024-04-25T11:30:50.614 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5036": { "addr": "172.21.15.157:6837/3688751078", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3688751078, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3688751078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5036, "incarnation": 44, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:30:48.888045+0000", "qdb_cluster": [ 4303, 5238, 5385, 5517 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 5036, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:30:52.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:30:52.483 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:30:52.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:52.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.791+0000 7fb978ba2640 1 Processor -- start 2024-04-25T11:30:52.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.791+0000 7fb978ba2640 1 -- start start 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb978ba2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb97415c0d0 0x7fb9740cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb978ba2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb974160370 0x7fb974166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb978ba2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97415c6f0 con 0x7fb97415ff60 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb978ba2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb9740c6310 con 0x7fb97415c0d0 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb978ba2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb97415c990 con 0x7fb974160370 2024-04-25T11:30:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb97415c0d0 0x7fb9740cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb97415c0d0 0x7fb9740cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54046/0 (socket says 172.21.15.136:54046) 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 -- 172.21.15.136:0/957760489 learned_addr learned my addr 172.21.15.136:0/957760489 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb972d9d640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb974160370 0x7fb974166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 -- 172.21.15.136:0/957760489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb974160370 msgr2=0x7fb974166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb974160370 0x7fb974166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 -- 172.21.15.136:0/957760489 >> v1:172.21.15.157:6789/0 conn(0x7fb97415ff60 legacy=0x7fb9740ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:52.784 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 -- 172.21.15.136:0/957760489 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97415cc30 con 0x7fb97415c0d0 2024-04-25T11:30:52.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb97259c640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb97415c0d0 0x7fb9740cb720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb964009870 tx=0x7fb96401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130b6866d7b937f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:52.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.795+0000 7fb971d9b640 1 -- 172.21.15.136:0/957760489 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9640b5070 con 0x7fb97415c0d0 2024-04-25T11:30:52.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb971d9b640 1 -- 172.21.15.136:0/957760489 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9640a8a50 con 0x7fb97415c0d0 2024-04-25T11:30:52.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb971d9b640 1 -- 172.21.15.136:0/957760489 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9640a8e20 con 0x7fb97415c0d0 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- 172.21.15.136:0/957760489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb97415c0d0 msgr2=0x7fb9740cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb97415c0d0 0x7fb9740cb720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb964009870 tx=0x7fb96401ed50 comp rx=0 tx=0).stop 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- 172.21.15.136:0/957760489 shutdown_connections 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb974160370 0x7fb974166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 --2- 172.21.15.136:0/957760489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb97415c0d0 0x7fb9740cb720 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- 172.21.15.136:0/957760489 >> 172.21.15.136:0/957760489 conn(0x7fb97406f9d0 msgr2=0x7fb97406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- 172.21.15.136:0/957760489 shutdown_connections 2024-04-25T11:30:52.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- 172.21.15.136:0/957760489 wait complete. 2024-04-25T11:30:52.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 Processor -- start 2024-04-25T11:30:52.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.799+0000 7fb978ba2640 1 -- start start 2024-04-25T11:30:52.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb97415c0d0 0x7fb9741e6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 0x7fb9740cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb9740cbc60 con 0x7fb974160370 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb97415fa40 con 0x7fb97415ff60 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb97415cf10 con 0x7fb97415c0d0 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54060/0 (socket says 172.21.15.136:54060) 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 -- 172.21.15.136:0/1261174894 learned_addr learned my addr 172.21.15.136:0/1261174894 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb97259c640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb97415c0d0 0x7fb9741e6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb96bfff640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 0x7fb9740cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 -- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb97415c0d0 msgr2=0x7fb9741e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb97415c0d0 0x7fb9741e6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:52.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 -- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 msgr2=0x7fb9740cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 0x7fb9740cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 -- 172.21.15.136:0/1261174894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9740cc610 con 0x7fb974160370 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb96bfff640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 0x7fb9740cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb972d9d640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb96c007d70 tx=0x7fb96c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c014420 con 0x7fb974160370 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96c019070 con 0x7fb974160370 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb974154080 con 0x7fb974160370 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c014820 con 0x7fb974160370 2024-04-25T11:30:52.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9741e55c0 con 0x7fb974160370 2024-04-25T11:30:52.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.803+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9740cc7e0 con 0x7fb974160370 2024-04-25T11:30:52.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.807+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb96c0149c0 con 0x7fb974160370 2024-04-25T11:30:52.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.807+0000 7fb96affd640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 0x7fb954071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:52.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.807+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fb96c08ffe0 con 0x7fb974160370 2024-04-25T11:30:52.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.807+0000 7fb97259c640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 0x7fb954071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:52.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.807+0000 7fb97259c640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 0x7fb954071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b2000 tx=0x7fb964008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:52.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:52.815+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb96c05c9b0 con 0x7fb974160370 2024-04-25T11:30:53.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb97406a350 con 0x7fb974160370 2024-04-25T11:30:53.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb96affd640 1 -- 172.21.15.136:0/1261174894 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6096 (secure 0 0 0) 0x7fb96c05c2b0 con 0x7fb974160370 2024-04-25T11:30:53.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:53.035 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:48.888045+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:30:53.036 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 44 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 msgr2=0x7fb954071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 0x7fb954071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb9640b2000 tx=0x7fb964008040 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 msgr2=0x7fb9740c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb96c007d70 tx=0x7fb96c00e4b0 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 shutdown_connections 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb974160370 0x7fb9740c7d00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb97415ff60 0x7fb9740cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb95406f400 0x7fb954071910 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 --2- 172.21.15.136:0/1261174894 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb97415c0d0 0x7fb9741e6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 >> 172.21.15.136:0/1261174894 conn(0x7fb97406f9d0 msgr2=0x7fb97406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 shutdown_connections 2024-04-25T11:30:53.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:53.046+0000 7fb978ba2640 1 -- 172.21.15.136:0/1261174894 wait complete. 2024-04-25T11:30:53.055 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:55.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:55.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 Processor -- start 2024-04-25T11:30:55.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 -- start start 2024-04-25T11:30:55.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feaac15a430 0x7feaac15a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaac15d750 con 0x7feaac0b5060 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7feaac15da30 con 0x7feaac15a430 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feab15b2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feaac15dd10 con 0x7feaac0ca290 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feaac15a430 0x7feaac15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feaac15a430 0x7feaac15a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35360/0 (socket says 172.21.15.136:35360) 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 -- 172.21.15.136:0/3168813403 learned_addr learned my addr 172.21.15.136:0/3168813403 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaaaffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7feaac0b5060 0x7feaac0c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:37366/0 (socket says 172.21.15.136:37366) 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaa3fff640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 -- 172.21.15.136:0/3168813403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 msgr2=0x7feaac0ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac0ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 -- 172.21.15.136:0/3168813403 >> v1:172.21.15.136:6789/0 conn(0x7feaac0b5060 legacy=0x7feaac0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaab7fe640 1 -- 172.21.15.136:0/3168813403 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac15dff0 con 0x7feaac15a430 2024-04-25T11:30:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaaa7fc640 1 -- 172.21.15.136:0/3168813403 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264876172 0 0) 0x7feaac15d750 con 0x7feaac0b5060 2024-04-25T11:30:55.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.230+0000 7feaa3fff640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:30:55.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feaab7fe640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac15a430 0x7feaac15a840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009e50 tx=0x7feaa401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3fc28360f8d84c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:55.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feaaa7fc640 1 -- 172.21.15.136:0/3168813403 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa40b4070 con 0x7feaac15a430 2024-04-25T11:30:55.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feaaa7fc640 1 -- 172.21.15.136:0/3168813403 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa40af400 con 0x7feaac15a430 2024-04-25T11:30:55.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feaaa7fc640 1 -- 172.21.15.136:0/3168813403 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa40af7d0 con 0x7feaac15a430 2024-04-25T11:30:55.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 -- 172.21.15.136:0/3168813403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac15a430 msgr2=0x7feaac15a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac15a430 0x7feaac15a840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009e50 tx=0x7feaa401f210 comp rx=0 tx=0).stop 2024-04-25T11:30:55.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 -- 172.21.15.136:0/3168813403 shutdown_connections 2024-04-25T11:30:55.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac15a430 0x7feaac15a840 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 --2- 172.21.15.136:0/3168813403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac0ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 -- 172.21.15.136:0/3168813403 >> 172.21.15.136:0/3168813403 conn(0x7feaac070f70 msgr2=0x7feaac068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:55.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 -- 172.21.15.136:0/3168813403 shutdown_connections 2024-04-25T11:30:55.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.234+0000 7feab15b2640 1 -- 172.21.15.136:0/3168813403 wait complete. 2024-04-25T11:30:55.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 Processor -- start 2024-04-25T11:30:55.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 -- start start 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac1efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feaac15a430 0x7feaac0b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feaac15ad80 con 0x7feaac0b5060 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feaac159f10 con 0x7feaac15a430 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feab15b2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feaac15d3f0 con 0x7feaac0ca290 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35374/0 (socket says 172.21.15.136:35374) 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 -- 172.21.15.136:0/283947886 learned_addr learned my addr 172.21.15.136:0/283947886 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaa3fff640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac1efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaab7fe640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feaac15a430 0x7feaac0b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 -- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 msgr2=0x7feaac1efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac1efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 -- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feaac15a430 msgr2=0x7feaac0b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feaac15a430 0x7feaac0b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.238+0000 7feaaaffd640 1 -- 172.21.15.136:0/283947886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac1f0380 con 0x7feaac0b5060 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaaaffd640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c4b0 tx=0x7fea9c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:55.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c018070 con 0x7feaac0b5060 2024-04-25T11:30:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9c007500 con 0x7feaac0b5060 2024-04-25T11:30:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c004a30 con 0x7feaac0b5060 2024-04-25T11:30:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac15b030 con 0x7feaac0b5060 2024-04-25T11:30:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac0ce930 con 0x7feaac0b5060 2024-04-25T11:30:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac06a350 con 0x7feaac0b5060 2024-04-25T11:30:55.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea9c01b070 con 0x7feaac0b5060 2024-04-25T11:30:55.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.242+0000 7feaa37fe640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 0x7fea8c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:55.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.246+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fea9c08e9f0 con 0x7feaac0b5060 2024-04-25T11:30:55.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.246+0000 7feaa3fff640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 0x7fea8c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:55.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.246+0000 7feaa3fff640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 0x7fea8c071910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:55.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.250+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fea9c05b450 con 0x7feaac0b5060 2024-04-25T11:30:55.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.470+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feaac0cadd0 con 0x7feaac0b5060 2024-04-25T11:30:55.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.470+0000 7feaa37fe640 1 -- 172.21.15.136:0/283947886 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6097 (secure 0 0 0) 0x7fea9c05ad50 con 0x7feaac0b5060 2024-04-25T11:30:55.461 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:55.461 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:53.901559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:resolve","state_seq":7,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303]},"id":1}]} 2024-04-25T11:30:55.461 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 45 2024-04-25T11:30:55.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 msgr2=0x7fea8c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 0x7fea8c071910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94008040 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 msgr2=0x7feaac1499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00c4b0 tx=0x7fea9c00c970 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 shutdown_connections 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feaac15a430 0x7feaac0b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea8c06f400 0x7fea8c071910 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feaac0ca290 0x7feaac1efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 --2- 172.21.15.136:0/283947886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaac0b5060 0x7feaac1499d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 >> 172.21.15.136:0/283947886 conn(0x7feaac070f70 msgr2=0x7feaac0c4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 shutdown_connections 2024-04-25T11:30:55.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:55.474+0000 7feab15b2640 1 -- 172.21.15.136:0/283947886 wait complete. 2024-04-25T11:30:55.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:57.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:30:57.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 Processor -- start 2024-04-25T11:30:57.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 -- start start 2024-04-25T11:30:57.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6de015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6de00cdee0 0x7f6de00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de00c6470 con 0x7f6de00c66e0 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6de00cd390 con 0x7f6de00cdee0 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de7449640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6de015c6e0 con 0x7f6de0156380 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de51e5640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6de00c66e0 0x7f6de015f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:37374/0 (socket says 172.21.15.136:37374) 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de51e5640 1 -- 172.21.15.136:0/2802342698 learned_addr learned my addr 172.21.15.136:0/2802342698 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de59e6640 1 --2- 172.21.15.136:0/2802342698 >> v2:172.21.15.157:3300/0 conn(0x7f6de00cdee0 0x7f6de00ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2313405465 0 0) 0x7f6de00c6470 con 0x7f6de00c66e0 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6de49e4640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6de015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.662+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dcc003530 con 0x7f6de00c66e0 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 65533105 0 0) 0x7f6dcc003530 con 0x7f6de00c66e0 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 msgr2=0x7f6de015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6de015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 >> v2:172.21.15.157:3300/0 conn(0x7f6de00cdee0 msgr2=0x7f6de00ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 --2- 172.21.15.136:0/2802342698 >> v2:172.21.15.157:3300/0 conn(0x7f6de00cdee0 0x7f6de00ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de0156790 con 0x7f6de00c66e0 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f6ddc0016a0 con 0x7f6de00c66e0 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 >> v1:172.21.15.136:6789/0 conn(0x7f6de00c66e0 legacy=0x7f6de015f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6dcc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dcc003f40 0x7f6dcc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dcc004d50 0x7f6dcc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6de015c6e0 con 0x7f6dcc004d50 2024-04-25T11:30:57.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6dcc00b9d0 con 0x7f6dcc003f40 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6dcc00bba0 con 0x7f6de0156380 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dcc004d50 0x7f6dcc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de49e4640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dcc003f40 0x7f6dcc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de51e5640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6dcc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 -- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 msgr2=0x7f6dcc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de0156380 0x7f6dcc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 -- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dcc003f40 msgr2=0x7f6dcc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dcc003f40 0x7f6dcc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 -- 172.21.15.136:0/2802342698 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dcc003530 con 0x7f6dcc004d50 2024-04-25T11:30:57.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de49e4640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dcc003f40 0x7f6dcc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:30:57.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6de59e6640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dcc004d50 0x7f6dcc005160 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009f90 tx=0x7f6dd801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46893b4c9d049890 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:57.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80b2070 con 0x7f6dcc004d50 2024-04-25T11:30:57.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.666+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd80a5c90 con 0x7f6dcc004d50 2024-04-25T11:30:57.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6dd7fff640 1 -- 172.21.15.136:0/2802342698 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8009c10 con 0x7f6dcc004d50 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- 172.21.15.136:0/2802342698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dcc004d50 msgr2=0x7f6dcc005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 --2- 172.21.15.136:0/2802342698 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dcc004d50 0x7f6dcc005160 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009f90 tx=0x7f6dd801cc80 comp rx=0 tx=0).stop 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de59e6640 1 -- 172.21.15.136:0/2802342698 reap_dead start 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- 172.21.15.136:0/2802342698 shutdown_connections 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- 172.21.15.136:0/2802342698 >> 172.21.15.136:0/2802342698 conn(0x7f6de006d6c0 msgr2=0x7f6de006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- 172.21.15.136:0/2802342698 shutdown_connections 2024-04-25T11:30:57.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- 172.21.15.136:0/2802342698 wait complete. 2024-04-25T11:30:57.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 Processor -- start 2024-04-25T11:30:57.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.670+0000 7f6de7449640 1 -- start start 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de015ed60 0x7f6de01f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de01f3020 0x7f6de01f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6de01601c0 con 0x7f6de015b7c0 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6de015fee0 con 0x7f6de01f3020 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6de01604a0 con 0x7f6de015ed60 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35392/0 (socket says 172.21.15.136:35392) 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 -- 172.21.15.136:0/79206733 learned_addr learned my addr 172.21.15.136:0/79206733 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de51e5640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de015ed60 0x7f6de01f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de49e4640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de01f3020 0x7f6de01f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 -- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de015ed60 msgr2=0x7f6de01f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de015ed60 0x7f6de01f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 -- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de01f3020 msgr2=0x7f6de01f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de01f3020 0x7f6de01f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 -- 172.21.15.136:0/79206733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de015f210 con 0x7f6de015b7c0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de59e6640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80b0040 tx=0x7f6dd8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80b2040 con 0x7f6de015b7c0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd80a5a90 con 0x7f6de015b7c0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de015f4f0 con 0x7f6de015b7c0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8009c10 con 0x7f6de015b7c0 2024-04-25T11:30:57.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de01f2820 con 0x7f6de015b7c0 2024-04-25T11:30:57.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.674+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de01f2330 con 0x7f6de015b7c0 2024-04-25T11:30:57.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.678+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6dd80bf060 con 0x7f6de015b7c0 2024-04-25T11:30:57.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.678+0000 7f6dd5ffb640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 0x7f6dc4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:30:57.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.678+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f6dd8129440 con 0x7f6de015b7c0 2024-04-25T11:30:57.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.678+0000 7f6de51e5640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 0x7f6dc4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:30:57.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.678+0000 7f6de51e5640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 0x7f6dc4071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c10 tx=0x7f6ddc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:30:57.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.682+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6dd8016990 con 0x7f6de015b7c0 2024-04-25T11:30:57.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6de006a350 con 0x7f6de015b7c0 2024-04-25T11:30:57.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6dd5ffb640 1 -- 172.21.15.136:0/79206733 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6097 (secure 0 0 0) 0x7f6dd80f5e60 con 0x7f6de015b7c0 2024-04-25T11:30:57.903 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:30:57.903 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:53.901559+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:resolve","state_seq":7,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303]},"id":1}]} 2024-04-25T11:30:57.903 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 45 2024-04-25T11:30:57.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 msgr2=0x7f6dc4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 0x7f6dc4071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc000c10 tx=0x7f6ddc009040 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 msgr2=0x7f6de0150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80b0040 tx=0x7f6dd8009750 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 shutdown_connections 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de015b7c0 0x7f6de0150c50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6de01f3020 0x7f6de01f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6dc406f400 0x7f6dc4071910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 --2- 172.21.15.136:0/79206733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6de015ed60 0x7f6de01f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 >> 172.21.15.136:0/79206733 conn(0x7f6de006d6c0 msgr2=0x7f6de01530a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 shutdown_connections 2024-04-25T11:30:57.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:30:57.914+0000 7f6de7449640 1 -- 172.21.15.136:0/79206733 wait complete. 2024-04-25T11:30:57.927 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:30:59.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:00.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 Processor -- start 2024-04-25T11:31:00.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 -- start start 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa90c156330 0x7fa90c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa90c0b5ba0 0x7fa90c0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c15d010 con 0x7fa90c0cdee0 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c1602f0 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91447e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c0ccb80 con 0x7fa90c156330 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa911a19640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa90c0cdee0 0x7fa90c0b5490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:37384/0 (socket says 172.21.15.136:37384) 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa911a19640 1 -- 172.21.15.136:0/2141636183 learned_addr learned my addr 172.21.15.136:0/2141636183 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa912a1b640 1 --2- 172.21.15.136:0/2141636183 >> v2:172.21.15.136:3300/0 conn(0x7fa90c0b5ba0 0x7fa90c0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.110+0000 7fa91221a640 1 --2- 172.21.15.136:0/2141636183 >> v2:172.21.15.157:3300/0 conn(0x7fa90c156330 0x7fa90c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa911218640 1 -- 172.21.15.136:0/2141636183 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229721664 0 0) 0x7fa8fc001120 con 0x7fa90c0cdee0 2024-04-25T11:31:00.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa912a1b640 1 -- 172.21.15.136:0/2141636183 >> v2:172.21.15.157:3300/0 conn(0x7fa90c156330 msgr2=0x7fa90c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:00.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa912a1b640 1 --2- 172.21.15.136:0/2141636183 >> v2:172.21.15.157:3300/0 conn(0x7fa90c156330 0x7fa90c15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa912a1b640 1 -- 172.21.15.136:0/2141636183 >> v1:172.21.15.136:6789/0 conn(0x7fa90c0cdee0 legacy=0x7fa90c0b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa912a1b640 1 -- 172.21.15.136:0/2141636183 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c156740 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa912a1b640 1 --2- 172.21.15.136:0/2141636183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c0b5ba0 0x7fa90c0ce2f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0cf010 tx=0x7fa90401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320822749ffc49cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:00.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa911218640 1 -- 172.21.15.136:0/2141636183 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9040b4070 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa911218640 1 -- 172.21.15.136:0/2141636183 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9040af600 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa911218640 1 -- 172.21.15.136:0/2141636183 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9040af9d0 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 -- 172.21.15.136:0/2141636183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c0b5ba0 msgr2=0x7fa90c0ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 --2- 172.21.15.136:0/2141636183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c0b5ba0 0x7fa90c0ce2f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0cf010 tx=0x7fa90401f110 comp rx=0 tx=0).stop 2024-04-25T11:31:00.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 -- 172.21.15.136:0/2141636183 shutdown_connections 2024-04-25T11:31:00.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 --2- 172.21.15.136:0/2141636183 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c0b5ba0 0x7fa90c0ce2f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 --2- 172.21.15.136:0/2141636183 >> v2:172.21.15.157:3300/0 conn(0x7fa90c156330 0x7fa90c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 -- 172.21.15.136:0/2141636183 >> 172.21.15.136:0/2141636183 conn(0x7fa90c06d6c0 msgr2=0x7fa90c06e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:00.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.114+0000 7fa91447e640 1 -- 172.21.15.136:0/2141636183 shutdown_connections 2024-04-25T11:31:00.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.118+0000 7fa91447e640 1 -- 172.21.15.136:0/2141636183 wait complete. 2024-04-25T11:31:00.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.118+0000 7fa91447e640 1 Processor -- start 2024-04-25T11:31:00.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- start start 2024-04-25T11:31:00.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa90c0b5ba0 0x7fa90c1f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa90c0cdee0 0x7fa90c14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c15fca0 con 0x7fa90c156330 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c15f9c0 con 0x7fa90c0cdee0 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c15ff80 con 0x7fa90c0b5ba0 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35402/0 (socket says 172.21.15.136:35402) 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 -- 172.21.15.136:0/1222015911 learned_addr learned my addr 172.21.15.136:0/1222015911 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa911a19640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa90c0cdee0 0x7fa90c14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91221a640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa90c0b5ba0 0x7fa90c1f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 -- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa90c0b5ba0 msgr2=0x7fa90c1f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa90c0b5ba0 0x7fa90c1f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 -- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa90c0cdee0 msgr2=0x7fa90c14e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa90c0cdee0 0x7fa90c14e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 -- 172.21.15.136:0/1222015911 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c14e660 con 0x7fa90c156330 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa912a1b640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa90401cd40 tx=0x7fa90401beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9040b4040 con 0x7fa90c156330 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9040039a0 con 0x7fa90c156330 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904009450 con 0x7fa90c156330 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c15ed10 con 0x7fa90c156330 2024-04-25T11:31:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.122+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c15f1d0 con 0x7fa90c156330 2024-04-25T11:31:00.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c06a350 con 0x7fa90c156330 2024-04-25T11:31:00.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa9040c4060 con 0x7fa90c156330 2024-04-25T11:31:00.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa902ffd640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 0x7fa8f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:00.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fa90412d440 con 0x7fa90c156330 2024-04-25T11:31:00.115 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa91221a640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 0x7fa8f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:00.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.126+0000 7fa91221a640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 0x7fa8f0071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa908009ac0 tx=0x7fa908009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.134+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa904018c70 con 0x7fa90c156330 2024-04-25T11:31:00.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.350+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa90c0cd9c0 con 0x7fa90c156330 2024-04-25T11:31:00.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.350+0000 7fa902ffd640 1 -- 172.21.15.136:0/1222015911 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6099 (secure 0 0 0) 0x7fa9040f97c0 con 0x7fa90c156330 2024-04-25T11:31:00.341 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:00.341 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:59.247342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:31:00.342 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 46 2024-04-25T11:31:00.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 msgr2=0x7fa8f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 0x7fa8f0071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa908009ac0 tx=0x7fa908009190 comp rx=0 tx=0).stop 2024-04-25T11:31:00.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 msgr2=0x7fa90c0b5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:00.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa90401cd40 tx=0x7fa90401beb0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 shutdown_connections 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa90c156330 0x7fa90c0b5fb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa90c0cdee0 0x7fa90c14e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f006f400 0x7fa8f0071910 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 --2- 172.21.15.136:0/1222015911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa90c0b5ba0 0x7fa90c1f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 >> 172.21.15.136:0/1222015911 conn(0x7fa90c06d6c0 msgr2=0x7fa90c0c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 shutdown_connections 2024-04-25T11:31:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:00.354+0000 7fa91447e640 1 -- 172.21.15.136:0/1222015911 wait complete. 2024-04-25T11:31:00.371 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:31:02.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.534+0000 7f824f2bb640 1 Processor -- start 2024-04-25T11:31:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.534+0000 7f824f2bb640 1 -- start start 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824f2bb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8248156750 0x7f82480ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824f2bb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82480cfba0 0x7f824806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824f2bb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f824815c410 con 0x7f82480ca7d0 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824f2bb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f824815c6b0 con 0x7f82480cfba0 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824f2bb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f824815c990 con 0x7f8248156750 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82480cfba0 0x7f824806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82480cfba0 0x7f824806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35408/0 (socket says 172.21.15.136:35408) 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 -- 172.21.15.136:0/2588715979 learned_addr learned my addr 172.21.15.136:0/2588715979 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d057640 1 --2- 172.21.15.136:0/2588715979 >> v2:172.21.15.157:3300/0 conn(0x7f8248156750 0x7f82480ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 -- 172.21.15.136:0/2588715979 >> v2:172.21.15.157:3300/0 conn(0x7f8248156750 msgr2=0x7f82480ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 --2- 172.21.15.136:0/2588715979 >> v2:172.21.15.157:3300/0 conn(0x7f8248156750 0x7f82480ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 -- 172.21.15.136:0/2588715979 >> v1:172.21.15.157:6789/0 conn(0x7f82480ca7d0 legacy=0x7f82480cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.538+0000 7f824d858640 1 -- 172.21.15.136:0/2588715979 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824815cc70 con 0x7f82480cfba0 2024-04-25T11:31:02.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824d858640 1 --2- 172.21.15.136:0/2588715979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480cfba0 0x7f824806e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f82480d00d0 tx=0x7f824001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50615c810e7ea9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:02.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f823ffff640 1 -- 172.21.15.136:0/2588715979 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82400a56c0 con 0x7f82480cfba0 2024-04-25T11:31:02.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f823ffff640 1 -- 172.21.15.136:0/2588715979 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82400a5860 con 0x7f82480cfba0 2024-04-25T11:31:02.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f823ffff640 1 -- 172.21.15.136:0/2588715979 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82400a5c30 con 0x7f82480cfba0 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 -- 172.21.15.136:0/2588715979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480cfba0 msgr2=0x7f824806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2588715979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480cfba0 0x7f824806e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f82480d00d0 tx=0x7f824001c9f0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 -- 172.21.15.136:0/2588715979 shutdown_connections 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2588715979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480cfba0 0x7f824806e470 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2588715979 >> v2:172.21.15.157:3300/0 conn(0x7f8248156750 0x7f82480ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 -- 172.21.15.136:0/2588715979 >> 172.21.15.136:0/2588715979 conn(0x7f8248070f70 msgr2=0x7f8248068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 -- 172.21.15.136:0/2588715979 shutdown_connections 2024-04-25T11:31:02.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.542+0000 7f824f2bb640 1 -- 172.21.15.136:0/2588715979 wait complete. 2024-04-25T11:31:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 Processor -- start 2024-04-25T11:31:02.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- start start 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82480ca7d0 0x7f82481f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82480cfba0 0x7f82480cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f82481692c0 con 0x7f82480af800 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f824806da70 con 0x7f82480ca7d0 2024-04-25T11:31:02.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8248169440 con 0x7f82480cfba0 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35416/0 (socket says 172.21.15.136:35416) 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 -- 172.21.15.136:0/2350396412 learned_addr learned my addr 172.21.15.136:0/2350396412 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d057640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82480ca7d0 0x7f82481f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824c856640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82480cfba0 0x7f82480cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 -- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82480cfba0 msgr2=0x7f82480cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82480cfba0 0x7f82480cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 -- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82480ca7d0 msgr2=0x7f82481f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82480ca7d0 0x7f82481f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 -- 172.21.15.136:0/2350396412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82480ce250 con 0x7f82480af800 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824d858640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f82400af040 tx=0x7f8240019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82400bc020 con 0x7f82480af800 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82400b8670 con 0x7f82480af800 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82480b0020 con 0x7f82480af800 2024-04-25T11:31:02.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240009510 con 0x7f82480af800 2024-04-25T11:31:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.546+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82480b05e0 con 0x7f82480af800 2024-04-25T11:31:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824806a350 con 0x7f82480af800 2024-04-25T11:31:02.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f82400bf020 con 0x7f82480af800 2024-04-25T11:31:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f823dffb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 0x7f822c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f824012a7b0 con 0x7f82480af800 2024-04-25T11:31:02.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f824d057640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 0x7f822c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:02.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.550+0000 7f824d057640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 0x7f822c071910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f824400b910 tx=0x7f8244008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:02.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.554+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82400c6030 con 0x7f82480af800 2024-04-25T11:31:02.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.806+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82480cadd0 con 0x7f82480af800 2024-04-25T11:31:02.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.806+0000 7f823dffb640 1 -- 172.21.15.136:0/2350396412 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6099 (secure 0 0 0) 0x7f82400ff020 con 0x7f82480af800 2024-04-25T11:31:02.796 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:02.796 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:30:59.247342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5238,5385,5517]},"id":1}]} 2024-04-25T11:31:02.797 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 46 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 msgr2=0x7f822c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 0x7f822c071910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f824400b910 tx=0x7f8244008040 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 msgr2=0x7f824814a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f82400af040 tx=0x7f8240019790 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 shutdown_connections 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82480af800 0x7f824814a8a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f822c06f400 0x7f822c071910 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82480cfba0 0x7f82480cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 --2- 172.21.15.136:0/2350396412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82480ca7d0 0x7f82481f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 >> 172.21.15.136:0/2350396412 conn(0x7f8248070f70 msgr2=0x7f82480caff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:02.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 shutdown_connections 2024-04-25T11:31:02.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:02.810+0000 7f824f2bb640 1 -- 172.21.15.136:0/2350396412 wait complete. 2024-04-25T11:31:02.819 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4303": { "addr": "172.21.15.157:6839/1671830331", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 1671830331, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 1671830331, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4303, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 6 }, "gid_5036": { "addr": "172.21.15.157:6837/3688751078", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3688751078, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3688751078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5036, "incarnation": 44, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:reconnect", "state_seq": 9 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:30:59.247342+0000", "qdb_cluster": [ 4303, 5238, 5385, 5517 ], "qdb_leader": 4303, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303, "mds_1": 5036, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:31:04.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:04.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.986+0000 7f4edad68640 1 Processor -- start 2024-04-25T11:31:04.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.986+0000 7f4edad68640 1 -- start start 2024-04-25T11:31:04.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4ed40c8cd0 0x7f4ed414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4ed40ce260 0x7f4ed40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ca010 0x7f4ed40ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed40c8550 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed40cbad0 con 0x7f4ed40ce260 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4edad68640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed40c7c20 con 0x7f4ed40ca010 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4ed40c8cd0 0x7f4ed414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4ed40c8cd0 0x7f4ed414f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46222/0 (socket says 172.21.15.136:46222) 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 -- 172.21.15.136:0/1676464910 learned_addr learned my addr 172.21.15.136:0/1676464910 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed9305640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ca010 0x7f4ed40ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed3fff640 1 --2- 172.21.15.136:0/1676464910 >> v2:172.21.15.157:3300/0 conn(0x7f4ed40ce260 0x7f4ed40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 -- 172.21.15.136:0/1676464910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ca010 msgr2=0x7f4ed40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ca010 0x7f4ed40ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 -- 172.21.15.136:0/1676464910 >> v2:172.21.15.157:3300/0 conn(0x7f4ed40ce260 msgr2=0x7f4ed40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:04.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/1676464910 >> v2:172.21.15.157:3300/0 conn(0x7f4ed40ce260 0x7f4ed40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.990+0000 7f4ed8b04640 1 -- 172.21.15.136:0/1676464910 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed40c7f00 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed414f680 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406f980 tx=0x7f4ec80213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c988af6d335f5fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:04.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4ed37fe640 1 -- 172.21.15.136:0/1676464910 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80b6070 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4ed37fe640 1 -- 172.21.15.136:0/1676464910 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec80b1600 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4ed37fe640 1 -- 172.21.15.136:0/1676464910 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80b19d0 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 -- 172.21.15.136:0/1676464910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 msgr2=0x7f4ed414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed414f680 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406f980 tx=0x7f4ec80213e0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 -- 172.21.15.136:0/1676464910 shutdown_connections 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ca010 0x7f4ed40ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 --2- 172.21.15.136:0/1676464910 >> v2:172.21.15.157:3300/0 conn(0x7f4ed40ce260 0x7f4ed40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 --2- 172.21.15.136:0/1676464910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed414f680 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 -- 172.21.15.136:0/1676464910 >> 172.21.15.136:0/1676464910 conn(0x7f4ed406f9d0 msgr2=0x7f4ed406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 -- 172.21.15.136:0/1676464910 shutdown_connections 2024-04-25T11:31:04.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.994+0000 7f4edad68640 1 -- 172.21.15.136:0/1676464910 wait complete. 2024-04-25T11:31:04.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 Processor -- start 2024-04-25T11:31:04.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- start start 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed40ca010 0x7f4ed4149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ce260 0x7f4ed414a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed40cb720 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed40ca960 con 0x7f4ed40ca010 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed4154200 con 0x7f4ed40ce260 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46234/0 (socket says 172.21.15.136:46234) 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 -- 172.21.15.136:0/3269624345 learned_addr learned my addr 172.21.15.136:0/3269624345 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed3fff640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed40ca010 0x7f4ed4149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed9305640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ce260 0x7f4ed414a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 -- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ce260 msgr2=0x7f4ed414a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ce260 0x7f4ed414a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 -- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed40ca010 msgr2=0x7f4ed4149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed40ca010 0x7f4ed4149ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 -- 172.21.15.136:0/3269624345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed4153d30 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed8b04640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4ec801fa90 tx=0x7f4ec801fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80c1070 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec80b1e50 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec80b6070 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed414b7b0 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:04.998+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed414c0a0 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed406a350 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4ec80c4070 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4ed17fa640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 0x7f4eb8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:04.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f4ec812cfb0 con 0x7f4ed40c8cd0 2024-04-25T11:31:04.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4ed3fff640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 0x7f4eb8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:04.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.002+0000 7f4ed3fff640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 0x7f4eb8071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40cc660 tx=0x7f4ec0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:04.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.010+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4ec801af40 con 0x7f4ed40c8cd0 2024-04-25T11:31:05.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.222+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ed4156790 con 0x7f4ed40c8cd0 2024-04-25T11:31:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.222+0000 7f4ed17fa640 1 -- 172.21.15.136:0/3269624345 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6097 (secure 0 0 0) 0x7f4ec80f9970 con 0x7f4ed40c8cd0 2024-04-25T11:31:05.212 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:05.213 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:04.282597+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303]},"id":1}]} 2024-04-25T11:31:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 47 2024-04-25T11:31:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 msgr2=0x7f4eb8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 0x7f4eb8071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40cc660 tx=0x7f4ec0008040 comp rx=0 tx=0).stop 2024-04-25T11:31:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 msgr2=0x7f4ed41537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4ec801fa90 tx=0x7f4ec801fc20 comp rx=0 tx=0).stop 2024-04-25T11:31:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 shutdown_connections 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4eb806f400 0x7f4eb8071910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ed40ce260 0x7f4ed414a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ed40ca010 0x7f4ed4149ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 --2- 172.21.15.136:0/3269624345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ed40c8cd0 0x7f4ed41537f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 >> 172.21.15.136:0/3269624345 conn(0x7f4ed406f9d0 msgr2=0x7f4ed40cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 shutdown_connections 2024-04-25T11:31:05.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:05.226+0000 7f4edad68640 1 -- 172.21.15.136:0/3269624345 wait complete. 2024-04-25T11:31:05.235 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:31:07.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:07.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.422+0000 7f629c09a640 1 Processor -- start 2024-04-25T11:31:07.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.422+0000 7f629c09a640 1 -- start start 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629c09a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f629415ab40 0x7f62940ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629c09a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629415d750 con 0x7f62940b5060 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629c09a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629415da30 con 0x7f62940ca290 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629c09a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f629415dd10 con 0x7f629415ab40 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f62940b5060 0x7f62940c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33628/0 (socket says 172.21.15.136:33628) 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 -- 172.21.15.136:0/1322256236 learned_addr learned my addr 172.21.15.136:0/1322256236 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629a637640 1 --2- 172.21.15.136:0/1322256236 >> v2:172.21.15.157:3300/0 conn(0x7f629415ab40 0x7f62940ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194487507 0 0) 0x7f629415d750 con 0x7f62940b5060 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6280003530 con 0x7f62940b5060 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1391027546 0 0) 0x7f6280003530 con 0x7f62940b5060 2024-04-25T11:31:07.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 >> v2:172.21.15.157:3300/0 conn(0x7f629415ab40 msgr2=0x7f62940ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 --2- 172.21.15.136:0/1322256236 >> v2:172.21.15.157:3300/0 conn(0x7f629415ab40 0x7f62940ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 >> v1:172.21.15.157:6789/0 conn(0x7f62940ca290 legacy=0x7f629415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629415dff0 con 0x7f62940b5060 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f6290001680 con 0x7f62940b5060 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 >> v1:172.21.15.136:6789/0 conn(0x7f62940b5060 legacy=0x7f62940c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 0x7f6280003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6280004810 0x7f6280004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62800051d0 0x7f628000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f629415dd10 con 0x7f62800039c0 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f628000c160 con 0x7f62800051d0 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f628000c360 con 0x7f6280004810 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 0x7f6280003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629a637640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62800051d0 0x7f628000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299635640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6280004810 0x7f6280004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 -- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6280004810 msgr2=0x7f6280004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6280004810 0x7f6280004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 -- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62800051d0 msgr2=0x7f628000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62800051d0 0x7f628000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f6299e36640 1 -- 172.21.15.136:0/1322256236 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6280003530 con 0x7f62800039c0 2024-04-25T11:31:07.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.426+0000 7f629a637640 1 -- 172.21.15.136:0/1322256236 reap_dead start 2024-04-25T11:31:07.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f6299e36640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 0x7f6280003e30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f629000c210 tx=0x7f629001f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1deadeefd973bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:07.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290007d40 con 0x7f62800039c0 2024-04-25T11:31:07.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290005b00 con 0x7f62800039c0 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f6298e34640 1 -- 172.21.15.136:0/1322256236 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290007070 con 0x7f62800039c0 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 msgr2=0x7f6280003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 0x7f6280003e30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f629000c210 tx=0x7f629001f2f0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- 172.21.15.136:0/1322256236 shutdown_connections 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 --2- 172.21.15.136:0/1322256236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62800039c0 0x7f6280003e30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- 172.21.15.136:0/1322256236 >> 172.21.15.136:0/1322256236 conn(0x7f6294070f70 msgr2=0x7f6294068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- 172.21.15.136:0/1322256236 shutdown_connections 2024-04-25T11:31:07.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- 172.21.15.136:0/1322256236 wait complete. 2024-04-25T11:31:07.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 Processor -- start 2024-04-25T11:31:07.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.430+0000 7f629c09a640 1 -- start start 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 0x7f629406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62941f3200 0x7f629415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f629415d3f0 con 0x7f6294151150 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6294159f10 con 0x7f62941f3200 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62940ca6a0 con 0x7f62800039c0 2024-04-25T11:31:07.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46250/0 (socket says 172.21.15.136:46250) 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 -- 172.21.15.136:0/1489365735 learned_addr learned my addr 172.21.15.136:0/1489365735 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f6299635640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62941f3200 0x7f629415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f6299e36640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 0x7f629406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 -- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 msgr2=0x7f629406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 0x7f629406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 -- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62941f3200 msgr2=0x7f629415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62941f3200 0x7f629415bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 -- 172.21.15.136:0/1489365735 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62940cdfa0 con 0x7f6294151150 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f6299e36640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 0x7f629406e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629a637640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6294068fc0 tx=0x7f628c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628c011e40 con 0x7f6294151150 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f628c01b070 con 0x7f6294151150 2024-04-25T11:31:07.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62940ceeb0 con 0x7f6294151150 2024-04-25T11:31:07.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629406dcc0 con 0x7f6294151150 2024-04-25T11:31:07.426 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.434+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628c0039a0 con 0x7f6294151150 2024-04-25T11:31:07.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f628c003bc0 con 0x7f6294151150 2024-04-25T11:31:07.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629406a350 con 0x7f6294151150 2024-04-25T11:31:07.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f628a7fc640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 0x7f6278071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:07.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f628c08ff10 con 0x7f6294151150 2024-04-25T11:31:07.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f6299e36640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 0x7f6278071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:07.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.438+0000 7f6299e36640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 0x7f6278071940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62900af000 tx=0x7f62900aa000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:07.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.442+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f628c05c8e0 con 0x7f6294151150 2024-04-25T11:31:07.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.678+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62940cbd30 con 0x7f6294151150 2024-04-25T11:31:07.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.678+0000 7f628a7fc640 1 -- 172.21.15.136:0/1489365735 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6097 (secure 0 0 0) 0x7f628c05c1e0 con 0x7f6294151150 2024-04-25T11:31:07.669 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:07.670 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:04.282597+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5517,5385,5238,4303]},"id":1}]} 2024-04-25T11:31:07.670 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 47 2024-04-25T11:31:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 msgr2=0x7f6278071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 0x7f6278071940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62900af000 tx=0x7f62900aa000 comp rx=0 tx=0).stop 2024-04-25T11:31:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 msgr2=0x7f62940ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6294068fc0 tx=0x7f628c0115a0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 shutdown_connections 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6294151150 0x7f62940ce210 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62941f3200 0x7f629415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f627806f430 0x7f6278071940 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 --2- 172.21.15.136:0/1489365735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62800039c0 0x7f629406e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 >> 172.21.15.136:0/1489365735 conn(0x7f6294070f70 msgr2=0x7f62940c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 shutdown_connections 2024-04-25T11:31:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:07.682+0000 7f629c09a640 1 -- 172.21.15.136:0/1489365735 wait complete. 2024-04-25T11:31:07.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:31:09.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.862+0000 7fbfc771d640 1 Processor -- start 2024-04-25T11:31:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.862+0000 7fbfc771d640 1 -- start start 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc771d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 0x7fbfc00ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc771d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc015c6f0 con 0x7fbfc0160370 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc771d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc015c990 con 0x7fbfc015c0d0 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc771d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc00c6310 con 0x7fbfc015ff60 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc5cba640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbfc0160370 0x7fbfc0166a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33636/0 (socket says 172.21.15.136:33636) 2024-04-25T11:31:09.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc5cba640 1 -- 172.21.15.136:0/617890626 learned_addr learned my addr 172.21.15.136:0/617890626 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:31:09.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 0x7fbfc00ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689042625 0 0) 0x7fbfc015c6f0 con 0x7fbfc0160370 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfac003530 con 0x7fbfc0160370 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 231384894 0 0) 0x7fbfac003530 con 0x7fbfc0160370 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 msgr2=0x7fbfc00ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 0x7fbfc00ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 >> v1:172.21.15.157:6789/0 conn(0x7fbfc015c0d0 legacy=0x7fbfc00cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc015cc30 con 0x7fbfc0160370 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fbfb8001a00 con 0x7fbfc0160370 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 >> v1:172.21.15.136:6789/0 conn(0x7fbfc0160370 legacy=0x7fbfc0166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac004810 0x7fbfac004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 0x7fbfac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc00c6310 con 0x7fbfac0039f0 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac00ba80 con 0x7fbfac004810 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbfac00bc80 con 0x7fbfc015ff60 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac004810 0x7fbfac004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 -- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 msgr2=0x7fbfac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:09.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc015ff60 0x7fbfac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 -- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac004810 msgr2=0x7fbfac004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac004810 0x7fbfac004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc54b9640 1 -- 172.21.15.136:0/617890626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac003530 con 0x7fbfac0039f0 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.866+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfac004810 0x7fbfac004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfac003e20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0001860 tx=0x7fbfb001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593aed827293d159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:09.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00b0070 con 0x7fbfac0039f0 2024-04-25T11:31:09.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb00ab600 con 0x7fbfac0039f0 2024-04-25T11:31:09.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfb7fff640 1 -- 172.21.15.136:0/617890626 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0001460 con 0x7fbfac0039f0 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 -- 172.21.15.136:0/617890626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 msgr2=0x7fbfac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 --2- 172.21.15.136:0/617890626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfac003e20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0001860 tx=0x7fbfb001c520 comp rx=0 tx=0).stop 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc5cba640 1 -- 172.21.15.136:0/617890626 reap_dead start 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 -- 172.21.15.136:0/617890626 shutdown_connections 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 -- 172.21.15.136:0/617890626 >> 172.21.15.136:0/617890626 conn(0x7fbfc006f9d0 msgr2=0x7fbfc006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 -- 172.21.15.136:0/617890626 shutdown_connections 2024-04-25T11:31:09.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.870+0000 7fbfc771d640 1 -- 172.21.15.136:0/617890626 wait complete. 2024-04-25T11:31:09.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 Processor -- start 2024-04-25T11:31:09.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 -- start start 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfc00cce30 0x7fbfc0145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc01439f0 0x7fbfc0143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc00c9170 con 0x7fbfac0039f0 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc00cabc0 con 0x7fbfc00cce30 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc771d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc0151220 con 0x7fbfc01439f0 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46258/0 (socket says 172.21.15.136:46258) 2024-04-25T11:31:09.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 -- 172.21.15.136:0/3735704317 learned_addr learned my addr 172.21.15.136:0/3735704317 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc5cba640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc01439f0 0x7fbfc0143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfc00cce30 0x7fbfc0145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 -- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc01439f0 msgr2=0x7fbfc0143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc01439f0 0x7fbfc0143e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 -- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfc00cce30 msgr2=0x7fbfc0145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfc00cce30 0x7fbfc0145a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.874+0000 7fbfc54b9640 1 -- 172.21.15.136:0/3735704317 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc0150d50 con 0x7fbfac0039f0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfc54b9640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbfb001ca00 tx=0x7fbfac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00b0040 con 0x7fbfac0039f0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb00bb070 con 0x7fbfac0039f0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc0146690 con 0x7fbfac0039f0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0005070 con 0x7fbfac0039f0 2024-04-25T11:31:09.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc00cd430 con 0x7fbfac0039f0 2024-04-25T11:31:09.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc006a350 con 0x7fbfac0039f0 2024-04-25T11:31:09.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbfb0005210 con 0x7fbfac0039f0 2024-04-25T11:31:09.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 0x7fbfa4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:09.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fbfb0129720 con 0x7fbfac0039f0 2024-04-25T11:31:09.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.878+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 0x7fbfa4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:09.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.882+0000 7fbfc4cb8640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 0x7fbfa4071910 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009160 tx=0x7fbfbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:09.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:09.886+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbfb0016230 con 0x7fbfac0039f0 2024-04-25T11:31:10.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.118+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfc0153e00 con 0x7fbfac0039f0 2024-04-25T11:31:10.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfb5ffb640 1 -- 172.21.15.136:0/3735704317 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6102 (secure 0 0 0) 0x7fbfb00f60f0 con 0x7fbfac0039f0 2024-04-25T11:31:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:10.111 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:31:10.112 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 48 2024-04-25T11:31:10.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 msgr2=0x7fbfa4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:10.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 0x7fbfa4071910 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009160 tx=0x7fbfbc008040 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 msgr2=0x7fbfc0150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbfb001ca00 tx=0x7fbfac005840 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 shutdown_connections 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbfa406f400 0x7fbfa4071910 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfc01439f0 0x7fbfc0143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfc00cce30 0x7fbfc0145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 --2- 172.21.15.136:0/3735704317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfac0039f0 0x7fbfc0150810 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 >> 172.21.15.136:0/3735704317 conn(0x7fbfc006f9d0 msgr2=0x7fbfc0147cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 shutdown_connections 2024-04-25T11:31:10.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:10.122+0000 7fbfc771d640 1 -- 172.21.15.136:0/3735704317 wait complete. 2024-04-25T11:31:10.133 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=1, replacing gid=4297 2024-04-25T11:31:10.133 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.d 2024-04-25T11:31:13.515 INFO:tasks.ceph.mds.f.smithi157.stderr:2024-04-25T11:31:13.525+0000 7f6c4649a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:31:19.134 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-04-25T11:31:19.134 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T11:31:19.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T11:31:19.137 INFO:tasks.ceph.mds.d:Started 2024-04-25T11:31:19.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:19.242 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-04-25T11:31:19.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.322+0000 7f76ca4fc640 1 Processor -- start 2024-04-25T11:31:19.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.322+0000 7f76ca4fc640 1 -- start start 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76ca4fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f76c40c79e0 0x7f76c40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76ca4fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76c415a3d0 0x7f76c415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76ca4fc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c40c9af0 con 0x7f76c40c96e0 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76ca4fc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f76c406cc10 con 0x7f76c415a3d0 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76ca4fc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f76c40cb220 con 0x7f76c40c79e0 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76c415a3d0 0x7f76c415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f76c40c96e0 0x7f76c41540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53012/0 (socket says 172.21.15.136:53012) 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76c415a3d0 0x7f76c415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39168/0 (socket says 172.21.15.136:39168) 2024-04-25T11:31:19.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c3fff640 1 -- 172.21.15.136:0/3482082034 learned_addr learned my addr 172.21.15.136:0/3482082034 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c37fe640 1 --2- 172.21.15.136:0/3482082034 >> v2:172.21.15.157:3300/0 conn(0x7f76c40c79e0 0x7f76c40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 -- 172.21.15.136:0/3482082034 >> v2:172.21.15.157:3300/0 conn(0x7f76c40c79e0 msgr2=0x7f76c40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 --2- 172.21.15.136:0/3482082034 >> v2:172.21.15.157:3300/0 conn(0x7f76c40c79e0 0x7f76c40c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 -- 172.21.15.136:0/3482082034 >> v1:172.21.15.136:6789/0 conn(0x7f76c40c96e0 legacy=0x7f76c41540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 -- 172.21.15.136:0/3482082034 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c40cb500 con 0x7f76c415a3d0 2024-04-25T11:31:19.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c2ffd640 1 -- 172.21.15.136:0/3482082034 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502619753 0 0) 0x7f76c40c9af0 con 0x7f76c40c96e0 2024-04-25T11:31:19.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c8a99640 1 --2- 172.21.15.136:0/3482082034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c415a7e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f76c40c8990 tx=0x7f76bc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce1e053cf183056 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:19.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c2ffd640 1 -- 172.21.15.136:0/3482082034 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0b2070 con 0x7f76c415a3d0 2024-04-25T11:31:19.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c2ffd640 1 -- 172.21.15.136:0/3482082034 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc0a57f0 con 0x7f76c415a3d0 2024-04-25T11:31:19.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.326+0000 7f76c2ffd640 1 -- 172.21.15.136:0/3482082034 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0a5bc0 con 0x7f76c415a3d0 2024-04-25T11:31:19.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- 172.21.15.136:0/3482082034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 msgr2=0x7f76c415a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/3482082034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c415a7e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f76c40c8990 tx=0x7f76bc01c930 comp rx=0 tx=0).stop 2024-04-25T11:31:19.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- 172.21.15.136:0/3482082034 shutdown_connections 2024-04-25T11:31:19.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/3482082034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c415a7e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/3482082034 >> v2:172.21.15.157:3300/0 conn(0x7f76c40c79e0 0x7f76c40c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- 172.21.15.136:0/3482082034 >> 172.21.15.136:0/3482082034 conn(0x7f76c406f9d0 msgr2=0x7f76c406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:19.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- 172.21.15.136:0/3482082034 shutdown_connections 2024-04-25T11:31:19.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- 172.21.15.136:0/3482082034 wait complete. 2024-04-25T11:31:19.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 Processor -- start 2024-04-25T11:31:19.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.330+0000 7f76ca4fc640 1 -- start start 2024-04-25T11:31:19.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c40c79e0 0x7f76c4145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 0x7f76c40cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f76c40ced40 con 0x7f76c415a3d0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f76c406d440 con 0x7f76c40c96e0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f76c41f6e90 con 0x7f76c40c79e0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39176/0 (socket says 172.21.15.136:39176) 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 -- 172.21.15.136:0/1356419692 learned_addr learned my addr 172.21.15.136:0/1356419692 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c37fe640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 0x7f76c40cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c3fff640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c40c79e0 0x7f76c4145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 -- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c40c79e0 msgr2=0x7f76c4145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c40c79e0 0x7f76c4145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 -- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 msgr2=0x7f76c40cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 0x7f76c40cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 -- 172.21.15.136:0/1356419692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c4157230 con 0x7f76c415a3d0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c37fe640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 0x7f76c40cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c8a99640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f76bc019540 tx=0x7f76bc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0b2040 con 0x7f76c415a3d0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc0bf070 con 0x7f76c415a3d0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0b86b0 con 0x7f76c415a3d0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c4157510 con 0x7f76c415a3d0 2024-04-25T11:31:19.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c4145ca0 con 0x7f76c415a3d0 2024-04-25T11:31:19.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c406a350 con 0x7f76c415a3d0 2024-04-25T11:31:19.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.334+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f76bc0b8850 con 0x7f76c415a3d0 2024-04-25T11:31:19.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.338+0000 7f76c0ff9640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 0x7f76b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:19.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.338+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f76bc12a830 con 0x7f76c415a3d0 2024-04-25T11:31:19.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.338+0000 7f76c3fff640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 0x7f76b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:19.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.338+0000 7f76c3fff640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 0x7f76b0071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76b4000f70 tx=0x7f76b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:19.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.342+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f76bc016520 con 0x7f76c415a3d0 2024-04-25T11:31:19.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76c414ff30 con 0x7f76c415a3d0 2024-04-25T11:31:19.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76c0ff9640 1 -- 172.21.15.136:0/1356419692 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6102 (secure 0 0 0) 0x7f76bc0f7180 con 0x7f76c415a3d0 2024-04-25T11:31:19.571 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:19.571 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[4303,5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:31:19.571 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 48 2024-04-25T11:31:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 msgr2=0x7f76b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 0x7f76b0071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76b4000f70 tx=0x7f76b4009040 comp rx=0 tx=0).stop 2024-04-25T11:31:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 msgr2=0x7f76c4143ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f76bc019540 tx=0x7f76bc0196d0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 shutdown_connections 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76c415a3d0 0x7f76c4143ad0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c40c96e0 0x7f76c40cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f76b006f400 0x7f76b0071910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 --2- 172.21.15.136:0/1356419692 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c40c79e0 0x7f76c4145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 >> 172.21.15.136:0/1356419692 conn(0x7f76c406f9d0 msgr2=0x7f76c406d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 shutdown_connections 2024-04-25T11:31:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:19.582+0000 7f76ca4fc640 1 -- 172.21.15.136:0/1356419692 wait complete. 2024-04-25T11:31:19.593 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T11:31:21.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:31:21.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 Processor -- start 2024-04-25T11:31:21.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 -- start start 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f730015c070 0x7f730006d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73000b4480 con 0x7f730015c480 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f730015c990 con 0x7f73000b5810 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f7307c36640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f73000b4a90 con 0x7f730015c070 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73051d1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f730015c070 0x7f730006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73061d3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f730015c480 0x7f7300166910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53020/0 (socket says 172.21.15.136:53020) 2024-04-25T11:31:21.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73061d3640 1 -- 172.21.15.136:0/297931643 learned_addr learned my addr 172.21.15.136:0/297931643 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 160699414 0 0) 0x7f73000b4480 con 0x7f730015c480 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72ec003530 con 0x7f730015c480 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73051d1640 1 -- 172.21.15.136:0/297931643 >> v1:172.21.15.157:6789/0 conn(0x7f73000b5810 legacy=0x7f73000b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73051d1640 1 -- 172.21.15.136:0/297931643 >> v1:172.21.15.136:6789/0 conn(0x7f730015c480 legacy=0x7f7300166910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1127582621 0 0) 0x7f72ec003530 con 0x7f730015c480 2024-04-25T11:31:21.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.754+0000 7f73051d1640 1 -- 172.21.15.136:0/297931643 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730015cc70 con 0x7f730015c070 2024-04-25T11:31:21.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f73051d1640 1 --2- 172.21.15.136:0/297931643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f730015c070 0x7f730006d510 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f72f00097c0 tx=0x7f72f001ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fc1b9c5d037621 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:21.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f00b5070 con 0x7f730015c070 2024-04-25T11:31:21.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f00a8630 con 0x7f730015c070 2024-04-25T11:31:21.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f73049d0640 1 -- 172.21.15.136:0/297931643 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8001a20 con 0x7f730015c070 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- 172.21.15.136:0/297931643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f730015c070 msgr2=0x7f730006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 --2- 172.21.15.136:0/297931643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f730015c070 0x7f730006d510 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f72f00097c0 tx=0x7f72f001ebb0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- 172.21.15.136:0/297931643 shutdown_connections 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 --2- 172.21.15.136:0/297931643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f730015c070 0x7f730006d510 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- 172.21.15.136:0/297931643 >> 172.21.15.136:0/297931643 conn(0x7f73000743a0 msgr2=0x7f73000d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- 172.21.15.136:0/297931643 shutdown_connections 2024-04-25T11:31:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- 172.21.15.136:0/297931643 wait complete. 2024-04-25T11:31:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 Processor -- start 2024-04-25T11:31:21.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.758+0000 7f7307c36640 1 -- start start 2024-04-25T11:31:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 0x7f730015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f730015c480 0x7f730015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f730006e030 con 0x7f73000b5810 2024-04-25T11:31:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f730015cf50 con 0x7f730015c480 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f73000cbee0 con 0x7f730015c070 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39194/0 (socket says 172.21.15.136:39194) 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 -- 172.21.15.136:0/2889450618 learned_addr learned my addr 172.21.15.136:0/2889450618 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73051d1640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 0x7f730015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73061d3640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f730015c480 0x7f730015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 -- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 msgr2=0x7f730015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 0x7f730015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 -- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f730015c480 msgr2=0x7f730015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f730015c480 0x7f730015b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 -- 172.21.15.136:0/2889450618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730015b9a0 con 0x7f73000b5810 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73051d1640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 0x7f730015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f73059d2640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c3c0 tx=0x7f72fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc017070 con 0x7f73000b5810 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc0052d0 con 0x7f73000b5810 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730015ef50 con 0x7f73000b5810 2024-04-25T11:31:21.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc003840 con 0x7f73000b5810 2024-04-25T11:31:21.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730015f1f0 con 0x7f73000b5810 2024-04-25T11:31:21.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.762+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730006a350 con 0x7f73000b5810 2024-04-25T11:31:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.766+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f72fc01a070 con 0x7f73000b5810 2024-04-25T11:31:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.766+0000 7f72f67fc640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 0x7f72e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:31:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.766+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f72fc08eb40 con 0x7f73000b5810 2024-04-25T11:31:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.766+0000 7f73051d1640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 0x7f72e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:31:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.766+0000 7f73051d1640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 0x7f72e4071910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b2000 tx=0x7f72f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:31:21.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:21.770+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f72fc005480 con 0x7f73000b5810 2024-04-25T11:31:21.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73000cd5f0 con 0x7f73000b5810 2024-04-25T11:31:21.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f72f67fc640 1 -- 172.21.15.136:0/2889450618 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7f72fc05b510 con 0x7f73000b5810 2024-04-25T11:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:31:21.991 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 msgr2=0x7f72e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 0x7f72e4071910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b2000 tx=0x7f72f0008040 comp rx=0 tx=0).stop 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 msgr2=0x7f7300155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c3c0 tx=0x7f72fc00c880 comp rx=0 tx=0).stop 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 shutdown_connections 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f730015c480 0x7f730015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f72e406f400 0x7f72e4071910 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f730015c070 0x7f730015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 --2- 172.21.15.136:0/2889450618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73000b5810 0x7f7300155b20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:31:21.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 >> 172.21.15.136:0/2889450618 conn(0x7f73000743a0 msgr2=0x7f73000cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:31:21.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 shutdown_connections 2024-04-25T11:31:21.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:31:22.002+0000 7f7307c36640 1 -- 172.21.15.136:0/2889450618 wait complete. 2024-04-25T11:31:22.011 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-04-25T11:31:22.011 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 59 secs before thrashing 2024-04-25T11:31:22.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:31:22.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:31:52.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:31:52.677 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:32:21.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:21.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 Processor -- start 2024-04-25T11:32:21.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 -- start start 2024-04-25T11:32:21.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa55c0b5810 0x7fa55c0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa55c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55c0b4480 con 0x7fa55c15c480 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa55c0b4a90 con 0x7fa55c0b5810 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa561143640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa55c15c990 con 0x7fa55c15c070 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa55c15c480 0x7fa55c15f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:37018/0 (socket says 172.21.15.136:37018) 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 -- 172.21.15.136:0/2456672826 learned_addr learned my addr 172.21.15.136:0/2456672826 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55259c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa55c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55ad9d640 1 --2- 172.21.15.136:0/2456672826 >> v2:172.21.15.157:3300/0 conn(0x7fa55c0b5810 0x7fa55c0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264563146 0 0) 0x7fa55c0b4480 con 0x7fa55c15c480 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa548003530 con 0x7fa55c15c480 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3038321267 0 0) 0x7fa548003530 con 0x7fa55c15c480 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 msgr2=0x7fa55c06d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa55c06d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 >> v2:172.21.15.157:3300/0 conn(0x7fa55c0b5810 msgr2=0x7fa55c0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 --2- 172.21.15.136:0/2456672826 >> v2:172.21.15.157:3300/0 conn(0x7fa55c0b5810 0x7fa55c0b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55c15cc70 con 0x7fa55c15c480 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fa554001a00 con 0x7fa55c15c480 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 >> v1:172.21.15.136:6789/0 conn(0x7fa55c15c480 legacy=0x7fa55c15f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa5480039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa548003f40 0x7fa548004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa548004d50 0x7fa548005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c0b4a90 con 0x7fa548004d50 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c15c990 con 0x7fa548003f40 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa54800bab0 con 0x7fa55c15c070 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa548004d50 0x7fa548005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55259c640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa548003f40 0x7fa548004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55ad9d640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa5480039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 -- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 msgr2=0x7fa5480039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:21.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c15c070 0x7fa5480039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 -- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa548003f40 msgr2=0x7fa548004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:21.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa548003f40 0x7fa548004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.214+0000 7fa55b59e640 1 -- 172.21.15.136:0/2456672826 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548003530 con 0x7fa548004d50 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa55b59e640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa548004d50 0x7fa548005160 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa554002240 tx=0x7fa55401d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b79dbd68cc49a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55400a4e0 con 0x7fa548004d50 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5540ad650 con 0x7fa548004d50 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa55a59c640 1 -- 172.21.15.136:0/2456672826 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5540ada40 con 0x7fa548004d50 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 -- 172.21.15.136:0/2456672826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa548004d50 msgr2=0x7fa548005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 --2- 172.21.15.136:0/2456672826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa548004d50 0x7fa548005160 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa554002240 tx=0x7fa55401d380 comp rx=0 tx=0).stop 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa55b59e640 1 -- 172.21.15.136:0/2456672826 reap_dead start 2024-04-25T11:32:21.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 -- 172.21.15.136:0/2456672826 shutdown_connections 2024-04-25T11:32:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 -- 172.21.15.136:0/2456672826 >> 172.21.15.136:0/2456672826 conn(0x7fa55c0743a0 msgr2=0x7fa55c0d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 -- 172.21.15.136:0/2456672826 shutdown_connections 2024-04-25T11:32:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.218+0000 7fa561143640 1 -- 172.21.15.136:0/2456672826 wait complete. 2024-04-25T11:32:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 Processor -- start 2024-04-25T11:32:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- start start 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa55c15fe70 0x7fa55c150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c1510b0 0x7fa55c154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c06e030 con 0x7fa55c151670 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c15cf50 con 0x7fa55c15fe70 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa55c0b5460 con 0x7fa55c1510b0 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50890/0 (socket says 172.21.15.136:50890) 2024-04-25T11:32:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 -- 172.21.15.136:0/1897157864 learned_addr learned my addr 172.21.15.136:0/1897157864 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55259c640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa55c15fe70 0x7fa55c150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 -- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c1510b0 msgr2=0x7fa55c154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c1510b0 0x7fa55c154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 -- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa55c15fe70 msgr2=0x7fa55c150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa55c15fe70 0x7fa55c150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 -- 172.21.15.136:0/1897157864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55c1f64f0 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa55ad9d640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa54c011580 tx=0x7fa54c011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c015870 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa54c015a10 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55c1f2460 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c015dc0 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.222+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55c1f2a20 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55c06a350 con 0x7fa55c151670 2024-04-25T11:32:21.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa54c00f460 con 0x7fa55c151670 2024-04-25T11:32:21.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa553fff640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 0x7fa53c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fa54c0900e0 con 0x7fa55c151670 2024-04-25T11:32:21.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa55259c640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 0x7fa53c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.226+0000 7fa55259c640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 0x7fa53c071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0c7a70 tx=0x7fa544008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.230+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa54c019030 con 0x7fa55c151670 2024-04-25T11:32:21.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa55c0cd5f0 con 0x7fa55c151670 2024-04-25T11:32:21.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa553fff640 1 -- 172.21.15.136:0/1897157864 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7fa54c05cac0 con 0x7fa55c151670 2024-04-25T11:32:21.438 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:21.438 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:21.438 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 msgr2=0x7fa53c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 0x7fa53c071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0c7a70 tx=0x7fa544008040 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 msgr2=0x7fa55c153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa54c011580 tx=0x7fa54c011a40 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 shutdown_connections 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa53c06f400 0x7fa53c071910 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa55c1510b0 0x7fa55c154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa55c15fe70 0x7fa55c150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 --2- 172.21.15.136:0/1897157864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c151670 0x7fa55c153480 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 >> 172.21.15.136:0/1897157864 conn(0x7fa55c0743a0 msgr2=0x7fa55c1e53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.450+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 shutdown_connections 2024-04-25T11:32:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.454+0000 7fa561143640 1 -- 172.21.15.136:0/1897157864 wait complete. 2024-04-25T11:32:21.459 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-04-25T11:32:21.460 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-25T11:32:21.460 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:32:21.481 INFO:tasks.ceph.mds.f:Stopped 2024-04-25T11:32:21.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:21.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 Processor -- start 2024-04-25T11:32:21.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 -- start start 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f55d815a5f0 0x7f55d80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815aa00 0x7f55d80c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d80c3880 con 0x7f55d8154060 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f55d80c8fd0 con 0x7f55d815a5f0 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55dcd66640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55d806d9b0 con 0x7f55d815aa00 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55cffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f55d815a5f0 0x7f55d80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55d659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f55d8154060 0x7f55d80c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:37026/0 (socket says 172.21.15.136:37026) 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55d659c640 1 -- 172.21.15.136:0/99817504 learned_addr learned my addr 172.21.15.136:0/99817504 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55d6d9d640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815aa00 0x7f55d80c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55cffff640 1 -- 172.21.15.136:0/99817504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815aa00 msgr2=0x7f55d80c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:21.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55cffff640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815aa00 0x7f55d80c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55cffff640 1 -- 172.21.15.136:0/99817504 >> v1:172.21.15.136:6789/0 conn(0x7f55d8154060 legacy=0x7f55d80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55d5d9b640 1 -- 172.21.15.136:0/99817504 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502472079 0 0) 0x7f55d80c3880 con 0x7f55d8154060 2024-04-25T11:32:21.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.646+0000 7f55cffff640 1 -- 172.21.15.136:0/99817504 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d8154470 con 0x7f55d815a5f0 2024-04-25T11:32:21.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55cffff640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d815a5f0 0x7f55d80c71b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f55d8154640 tx=0x7f55c001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c8f891f74b4cb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55d5d9b640 1 -- 172.21.15.136:0/99817504 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b5070 con 0x7f55d815a5f0 2024-04-25T11:32:21.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55d5d9b640 1 -- 172.21.15.136:0/99817504 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00a8990 con 0x7f55d815a5f0 2024-04-25T11:32:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55d5d9b640 1 -- 172.21.15.136:0/99817504 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00a8d60 con 0x7f55d815a5f0 2024-04-25T11:32:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 -- 172.21.15.136:0/99817504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d815a5f0 msgr2=0x7f55d80c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d815a5f0 0x7f55d80c71b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f55d8154640 tx=0x7f55c001ec90 comp rx=0 tx=0).stop 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 -- 172.21.15.136:0/99817504 shutdown_connections 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815aa00 0x7f55d80c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 --2- 172.21.15.136:0/99817504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d815a5f0 0x7f55d80c71b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 -- 172.21.15.136:0/99817504 >> 172.21.15.136:0/99817504 conn(0x7f55d80ca5a0 msgr2=0x7f55d80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 -- 172.21.15.136:0/99817504 shutdown_connections 2024-04-25T11:32:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.650+0000 7f55dcd66640 1 -- 172.21.15.136:0/99817504 wait complete. 2024-04-25T11:32:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 Processor -- start 2024-04-25T11:32:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- start start 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815a5f0 0x7f55d80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55d815aa00 0x7f55d81461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f55d815a0d0 con 0x7f55d8154060 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55d806df30 con 0x7f55d815aa00 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55d8159d80 con 0x7f55d815a5f0 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50912/0 (socket says 172.21.15.136:50912) 2024-04-25T11:32:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 -- 172.21.15.136:0/121446674 learned_addr learned my addr 172.21.15.136:0/121446674 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55cffff640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815a5f0 0x7f55d80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d6d9d640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55d815aa00 0x7f55d81461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 -- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815a5f0 msgr2=0x7f55d80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815a5f0 0x7f55d80cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 -- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55d815aa00 msgr2=0x7f55d81461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55d815aa00 0x7f55d81461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 -- 172.21.15.136:0/121446674 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d80cbee0 con 0x7f55d8154060 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55d659c640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c4b0 tx=0x7f55c800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8018070 con 0x7f55d8154060 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c8007500 con 0x7f55d8154060 2024-04-25T11:32:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d80c5110 con 0x7f55d8154060 2024-04-25T11:32:21.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8004a30 con 0x7f55d8154060 2024-04-25T11:32:21.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.654+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d80c6440 con 0x7f55d8154060 2024-04-25T11:32:21.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d806a350 con 0x7f55d8154060 2024-04-25T11:32:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f55c801b070 con 0x7f55d8154060 2024-04-25T11:32:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55ceffd640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 0x7f55b8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f55c808eaa0 con 0x7f55d8154060 2024-04-25T11:32:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55cffff640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 0x7f55b8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.658+0000 7f55cffff640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 0x7f55b8071940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f55c00b2000 tx=0x7f55c00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.662+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f55c805b470 con 0x7f55d8154060 2024-04-25T11:32:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55d8075120 con 0x7f55d8154060 2024-04-25T11:32:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55ceffd640 1 -- 172.21.15.136:0/121446674 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7f55c805ad70 con 0x7f55d8154060 2024-04-25T11:32:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:21.886 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 msgr2=0x7f55b8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 0x7f55b8071940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f55c00b2000 tx=0x7f55c00aa070 comp rx=0 tx=0).stop 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 msgr2=0x7f55d8150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c4b0 tx=0x7f55c800c970 comp rx=0 tx=0).stop 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 shutdown_connections 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55d815aa00 0x7f55d81461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f55b806f430 0x7f55b8071940 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55d815a5f0 0x7f55d80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 --2- 172.21.15.136:0/121446674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55d8154060 0x7f55d8150700 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 >> 172.21.15.136:0/121446674 conn(0x7f55d80ca5a0 msgr2=0x7f55d8150cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 shutdown_connections 2024-04-25T11:32:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:21.898+0000 7f55dcd66640 1 -- 172.21.15.136:0/121446674 wait complete. 2024-04-25T11:32:21.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:22.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:32:22.799 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:32:23.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:24.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 Processor -- start 2024-04-25T11:32:24.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 -- start start 2024-04-25T11:32:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 0x7fb50c0c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb50c0c9570 0x7fb50c158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50c1596f0 con 0x7fb50c1592e0 2024-04-25T11:32:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb50c0c8370 con 0x7fb50c0c9570 2024-04-25T11:32:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb512f22640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb50c0c8090 con 0x7fb50c153190 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb50c0c9570 0x7fb50c158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb50c0c9570 0x7fb50c158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50914/0 (socket says 172.21.15.136:50914) 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 -- 172.21.15.136:0/1010913800 learned_addr learned my addr 172.21.15.136:0/1010913800 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb510cbe640 1 --2- 172.21.15.136:0/1010913800 >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 0x7fb50c0c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 -- 172.21.15.136:0/1010913800 >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 msgr2=0x7fb50c0c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 --2- 172.21.15.136:0/1010913800 >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 0x7fb50c0c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.082+0000 7fb50bfff640 1 -- 172.21.15.136:0/1010913800 >> v1:172.21.15.136:6789/0 conn(0x7fb50c1592e0 legacy=0x7fb50c15c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50bfff640 1 -- 172.21.15.136:0/1010913800 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c0c6aa0 con 0x7fb50c0c9570 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50b7fe640 1 -- 172.21.15.136:0/1010913800 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799603880 0 0) 0x7fb50c1596f0 con 0x7fb50c1592e0 2024-04-25T11:32:24.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb510cbe640 1 --2- 172.21.15.136:0/1010913800 >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 0x7fb50c0c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:24.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50bfff640 1 --2- 172.21.15.136:0/1010913800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c0c9570 0x7fb50c158da0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb50c074480 tx=0x7fb4f801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a854996df30ee19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:24.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50b7fe640 1 -- 172.21.15.136:0/1010913800 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80b5070 con 0x7fb50c0c9570 2024-04-25T11:32:24.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50b7fe640 1 -- 172.21.15.136:0/1010913800 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f80a8a10 con 0x7fb50c0c9570 2024-04-25T11:32:24.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb50b7fe640 1 -- 172.21.15.136:0/1010913800 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80a8de0 con 0x7fb50c0c9570 2024-04-25T11:32:24.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 -- 172.21.15.136:0/1010913800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c0c9570 msgr2=0x7fb50c158da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 --2- 172.21.15.136:0/1010913800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c0c9570 0x7fb50c158da0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb50c074480 tx=0x7fb4f801ed10 comp rx=0 tx=0).stop 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 -- 172.21.15.136:0/1010913800 shutdown_connections 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 --2- 172.21.15.136:0/1010913800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c0c9570 0x7fb50c158da0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 --2- 172.21.15.136:0/1010913800 >> v2:172.21.15.157:3300/0 conn(0x7fb50c153190 0x7fb50c0c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 -- 172.21.15.136:0/1010913800 >> 172.21.15.136:0/1010913800 conn(0x7fb50c0c9d50 msgr2=0x7fb50c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 -- 172.21.15.136:0/1010913800 shutdown_connections 2024-04-25T11:32:24.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.086+0000 7fb512f22640 1 -- 172.21.15.136:0/1010913800 wait complete. 2024-04-25T11:32:24.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 Processor -- start 2024-04-25T11:32:24.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- start start 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb50c1592e0 0x7fb50c13fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 0x7fb50c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c0c7340 con 0x7fb50c153190 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c0c7060 con 0x7fb50c1592e0 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb50c0ca870 con 0x7fb50c1438e0 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50924/0 (socket says 172.21.15.136:50924) 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 -- 172.21.15.136:0/105644893 learned_addr learned my addr 172.21.15.136:0/105644893 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb50bfff640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb50c1592e0 0x7fb50c13fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb5114bf640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 0x7fb50c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 -- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 msgr2=0x7fb50c1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 0x7fb50c1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 -- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb50c1592e0 msgr2=0x7fb50c13fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb50c1592e0 0x7fb50c13fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 -- 172.21.15.136:0/105644893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c1f4570 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb5114bf640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 0x7fb50c1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb510cbe640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb50000c7f0 tx=0x7fb50000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500015070 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5000077c0 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500007bb0 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c1f3b50 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c146bc0 con 0x7fb50c153190 2024-04-25T11:32:24.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.090+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50c06a350 con 0x7fb50c153190 2024-04-25T11:32:24.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.094+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb500018070 con 0x7fb50c153190 2024-04-25T11:32:24.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.094+0000 7fb5097fa640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 0x7fb4f0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:24.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.094+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fb50008fd00 con 0x7fb50c153190 2024-04-25T11:32:24.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.094+0000 7fb50bfff640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 0x7fb4f0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:24.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.094+0000 7fb50bfff640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 0x7fb4f0071940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80b2000 tx=0x7fb4f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:24.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.098+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb50005c720 con 0x7fb50c153190 2024-04-25T11:32:24.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb50c071300 con 0x7fb50c153190 2024-04-25T11:32:24.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb5097fa640 1 -- 172.21.15.136:0/105644893 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7fb50005c020 con 0x7fb50c153190 2024-04-25T11:32:24.287 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:24.288 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:24.288 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:24.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 msgr2=0x7fb4f0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 0x7fb4f0071940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb4f80b2000 tx=0x7fb4f8008040 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 msgr2=0x7fb50c1f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb50000c7f0 tx=0x7fb50000ccb0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 shutdown_connections 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb4f006f430 0x7fb4f0071940 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb50c1438e0 0x7fb50c1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb50c1592e0 0x7fb50c13fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 --2- 172.21.15.136:0/105644893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb50c153190 0x7fb50c1f4030 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 >> 172.21.15.136:0/105644893 conn(0x7fb50c0c9d50 msgr2=0x7fb50c06ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 shutdown_connections 2024-04-25T11:32:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:24.298+0000 7fb512f22640 1 -- 172.21.15.136:0/105644893 wait complete. 2024-04-25T11:32:24.305 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:26.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:26.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 Processor -- start 2024-04-25T11:32:26.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 -- start start 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7eec0c79e0 0x7f7eec0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7eec15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec0c9af0 con 0x7f7eec0c96e0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7eec06cc10 con 0x7f7eec0c79e0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef3b49640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7eec0cb220 con 0x7f7eec15a3d0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef18e5640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7eec0c96e0 0x7f7eec1540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59618/0 (socket says 172.21.15.136:59618) 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef18e5640 1 -- 172.21.15.136:0/837383684 learned_addr learned my addr 172.21.15.136:0/837383684 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/837383684 >> v2:172.21.15.157:3300/0 conn(0x7f7eec0c79e0 0x7f7eec0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef20e6640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7eec15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213526590 0 0) 0x7f7eec0c9af0 con 0x7f7eec0c96e0 2024-04-25T11:32:26.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.477+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed4003530 con 0x7f7eec0c96e0 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 275774943 0 0) 0x7f7ed4003530 con 0x7f7eec0c96e0 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 msgr2=0x7f7eec15a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7eec15a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 >> v2:172.21.15.157:3300/0 conn(0x7f7eec0c79e0 msgr2=0x7f7eec0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 --2- 172.21.15.136:0/837383684 >> v2:172.21.15.157:3300/0 conn(0x7f7eec0c79e0 0x7f7eec0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec0cb500 con 0x7f7eec0c96e0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f7ee4001a20 con 0x7f7eec0c96e0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 >> v1:172.21.15.136:6789/0 conn(0x7f7eec0c96e0 legacy=0x7f7eec1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7ed40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed4003f40 0x7f7ed4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed4004d50 0x7f7ed4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec06cc10 con 0x7f7ed4003f40 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec0cb220 con 0x7f7ed4004d50 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed400ba20 con 0x7f7eec15a3d0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed4003f40 0x7f7ed4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef18e5640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7ed40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef20e6640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed4004d50 0x7f7ed4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 -- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 msgr2=0x7f7ed40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eec15a3d0 0x7f7ed40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 -- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed4004d50 msgr2=0x7f7ed4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed4004d50 0x7f7ed4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 -- 172.21.15.136:0/837383684 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed4003530 con 0x7f7ed4003f40 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed4003f40 0x7f7ed4004810 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ab0 tx=0x7f7ee001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc99835052ae797 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00b2070 con 0x7f7ed4003f40 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee00a5570 con 0x7f7ed4003f40 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef08e3640 1 -- 172.21.15.136:0/837383684 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00096b0 con 0x7f7ed4003f40 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 -- 172.21.15.136:0/837383684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed4003f40 msgr2=0x7f7ed4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/837383684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ed4003f40 0x7f7ed4004810 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ab0 tx=0x7f7ee001ca90 comp rx=0 tx=0).stop 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef20e6640 1 -- 172.21.15.136:0/837383684 reap_dead start 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 -- 172.21.15.136:0/837383684 shutdown_connections 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 -- 172.21.15.136:0/837383684 >> 172.21.15.136:0/837383684 conn(0x7f7eec06f9d0 msgr2=0x7f7eec06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 -- 172.21.15.136:0/837383684 shutdown_connections 2024-04-25T11:32:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.481+0000 7f7ef3b49640 1 -- 172.21.15.136:0/837383684 wait complete. 2024-04-25T11:32:26.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 Processor -- start 2024-04-25T11:32:26.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 -- start start 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 0x7f7eec1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eec154870 0x7f7eec1484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec0ced40 con 0x7f7eec1f3c20 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec0cf0f0 con 0x7f7eec154870 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7eec148ef0 con 0x7f7ed4003f40 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37126/0 (socket says 172.21.15.136:37126) 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 -- 172.21.15.136:0/1752458247 learned_addr learned my addr 172.21.15.136:0/1752458247 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef10e4640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eec154870 0x7f7eec1484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef18e5640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 0x7f7eec1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 -- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 msgr2=0x7f7eec1556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 0x7f7eec1556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 -- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eec154870 msgr2=0x7f7eec1484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eec154870 0x7f7eec1484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 -- 172.21.15.136:0/1752458247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec148a20 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef18e5640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 0x7f7eec1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef20e6640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7eec075160 tx=0x7f7edc011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc019020 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc003620 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eec1498b0 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.485+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc003a10 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eec149e70 con 0x7f7eec1f3c20 2024-04-25T11:32:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eec1f4660 con 0x7f7eec1f3c20 2024-04-25T11:32:26.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7edc01c020 con 0x7f7eec1f3c20 2024-04-25T11:32:26.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7eea7fc640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 0x7f7ed8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:26.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f7edc08f920 con 0x7f7eec1f3c20 2024-04-25T11:32:26.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7ef18e5640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 0x7f7ed8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:26.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.489+0000 7f7ef18e5640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 0x7f7ed8071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4000f90 tx=0x7f7ee40014a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:26.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.493+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7edc023030 con 0x7f7eec1f3c20 2024-04-25T11:32:26.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.685+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7eec06a350 con 0x7f7eec1f3c20 2024-04-25T11:32:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.685+0000 7f7eea7fc640 1 -- 172.21.15.136:0/1752458247 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7f7edc05c380 con 0x7f7eec1f3c20 2024-04-25T11:32:26.675 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:26.675 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:26.675 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 msgr2=0x7f7ed8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 0x7f7ed8071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4000f90 tx=0x7f7ee40014a0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 msgr2=0x7f7eec13da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7eec075160 tx=0x7f7edc011920 comp rx=0 tx=0).stop 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 shutdown_connections 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec1f3c20 0x7f7eec13da90 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eec154870 0x7f7eec1484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7ed806f400 0x7f7ed8071910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 --2- 172.21.15.136:0/1752458247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7ed4003f40 0x7f7eec1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 >> 172.21.15.136:0/1752458247 conn(0x7f7eec06f9d0 msgr2=0x7f7eec0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 shutdown_connections 2024-04-25T11:32:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:26.689+0000 7f7ef3b49640 1 -- 172.21.15.136:0/1752458247 wait complete. 2024-04-25T11:32:26.694 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:28.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:28.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 Processor -- start 2024-04-25T11:32:28.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 -- start start 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f28b80ca290 0x7f28b80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b815a430 0x7f28b815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b815d750 con 0x7f28b80b5060 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f28b815da30 con 0x7f28b80ca290 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28be1de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f28b815dd10 con 0x7f28b815a430 2024-04-25T11:32:28.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f28b80ca290 0x7f28b80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f28b80ca290 0x7f28b80ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37142/0 (socket says 172.21.15.136:37142) 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 -- 172.21.15.136:0/3910958464 learned_addr learned my addr 172.21.15.136:0/3910958464 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b77fe640 1 --1- 172.21.15.136:0/3910958464 >> v1:172.21.15.136:6789/0 conn(0x7f28b80b5060 0x7f28b80c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59632/0 (socket says 172.21.15.136:59632) 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b7fff640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b815a430 0x7f28b815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 -- 172.21.15.136:0/3910958464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b815a430 msgr2=0x7f28b815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b815a430 0x7f28b815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 -- 172.21.15.136:0/3910958464 >> v1:172.21.15.136:6789/0 conn(0x7f28b80b5060 legacy=0x7f28b80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:28.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.865+0000 7f28b6ffd640 1 -- 172.21.15.136:0/3910958464 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b815dff0 con 0x7f28b80ca290 2024-04-25T11:32:28.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28b6ffd640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b80ca6a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009870 tx=0x7f28ac01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4aa6a5f4de927d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:28.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28b67fc640 1 -- 172.21.15.136:0/3910958464 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0b5070 con 0x7f28b80ca290 2024-04-25T11:32:28.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28b67fc640 1 -- 172.21.15.136:0/3910958464 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28ac0a8a50 con 0x7f28b80ca290 2024-04-25T11:32:28.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28b67fc640 1 -- 172.21.15.136:0/3910958464 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0a8e20 con 0x7f28b80ca290 2024-04-25T11:32:28.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 -- 172.21.15.136:0/3910958464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 msgr2=0x7f28b80ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:28.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b80ca6a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009870 tx=0x7f28ac01ed50 comp rx=0 tx=0).stop 2024-04-25T11:32:28.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 -- 172.21.15.136:0/3910958464 shutdown_connections 2024-04-25T11:32:28.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b815a430 0x7f28b815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:28.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 --2- 172.21.15.136:0/3910958464 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b80ca6a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:28.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 -- 172.21.15.136:0/3910958464 >> 172.21.15.136:0/3910958464 conn(0x7f28b8070f70 msgr2=0x7f28b8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:28.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 -- 172.21.15.136:0/3910958464 shutdown_connections 2024-04-25T11:32:28.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.869+0000 7f28be1de640 1 -- 172.21.15.136:0/3910958464 wait complete. 2024-04-25T11:32:28.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 Processor -- start 2024-04-25T11:32:28.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- start start 2024-04-25T11:32:28.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b80b5060 0x7f28b81f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28b815a430 0x7f28b81f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f28b815ad80 con 0x7f28b80ca290 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f28b8159f10 con 0x7f28b815a430 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f28b815d3f0 con 0x7f28b80b5060 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37148/0 (socket says 172.21.15.136:37148) 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 -- 172.21.15.136:0/508701882 learned_addr learned my addr 172.21.15.136:0/508701882 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b77fe640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b80b5060 0x7f28b81f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b7fff640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28b815a430 0x7f28b81f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 -- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b80b5060 msgr2=0x7f28b81f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b80b5060 0x7f28b81f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 -- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28b815a430 msgr2=0x7f28b81f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28b815a430 0x7f28b81f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:28.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 -- 172.21.15.136:0/508701882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b8153f10 con 0x7f28b80ca290 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28b6ffd640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0b3040 tx=0x7f28ac01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0c0070 con 0x7f28b80ca290 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28ac0ae790 con 0x7f28b80ca290 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0b5070 con 0x7f28b80ca290 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b81f30e0 con 0x7f28b80ca290 2024-04-25T11:32:28.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.873+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b815b630 con 0x7f28b80ca290 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b806a350 con 0x7f28b80ca290 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f28ac0c3070 con 0x7f28b80ca290 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f289ffff640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 0x7f28a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f28ac12d380 con 0x7f28b80ca290 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f28b77fe640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 0x7f28a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.877+0000 7f28b77fe640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 0x7f28a4071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f28a8000f70 tx=0x7f28a8007cf0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:28.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:28.881+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f28ac0188b0 con 0x7f28b80ca290 2024-04-25T11:32:29.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28b80cadd0 con 0x7f28b80ca290 2024-04-25T11:32:29.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f289ffff640 1 -- 172.21.15.136:0/508701882 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7f28ac0f9d50 con 0x7f28b80ca290 2024-04-25T11:32:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:29.062 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:29.062 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:29.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 msgr2=0x7f28a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 0x7f28a4071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f28a8000f70 tx=0x7f28a8007cf0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 msgr2=0x7f28b81481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:29.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0b3040 tx=0x7f28ac01baf0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 shutdown_connections 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28b815a430 0x7f28b81f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80ca290 0x7f28b81481e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f28a406f400 0x7f28a4071910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 --2- 172.21.15.136:0/508701882 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28b80b5060 0x7f28b81f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.073+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 >> 172.21.15.136:0/508701882 conn(0x7f28b8070f70 msgr2=0x7f28b814a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.077+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 shutdown_connections 2024-04-25T11:32:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:29.077+0000 7f28be1de640 1 -- 172.21.15.136:0/508701882 wait complete. 2024-04-25T11:32:29.081 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:31.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:31.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 Processor -- start 2024-04-25T11:32:31.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 -- start start 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba70153190 0x7fba700c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba700c9570 0x7fba70158da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba701596f0 con 0x7fba701592e0 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fba700c8370 con 0x7fba70153190 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba74904640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba700c8090 con 0x7fba700c9570 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba70153190 0x7fba700c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba70153190 0x7fba700c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37154/0 (socket says 172.21.15.136:37154) 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 -- 172.21.15.136:0/1585669664 learned_addr learned my addr 172.21.15.136:0/1585669664 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba67fff640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba700c9570 0x7fba70158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 -- 172.21.15.136:0/1585669664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba700c9570 msgr2=0x7fba70158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba700c9570 0x7fba70158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 -- 172.21.15.136:0/1585669664 >> v1:172.21.15.157:6789/0 conn(0x7fba701592e0 legacy=0x7fba7015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:31.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.257+0000 7fba6e59c640 1 -- 172.21.15.136:0/1585669664 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba700c6aa0 con 0x7fba70153190 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba6e59c640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba70153190 0x7fba700c9030 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cab842e8e5e126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba6dd9b640 1 -- 172.21.15.136:0/1585669664 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600b5070 con 0x7fba70153190 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba6dd9b640 1 -- 172.21.15.136:0/1585669664 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba600a89d0 con 0x7fba70153190 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba6dd9b640 1 -- 172.21.15.136:0/1585669664 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600a8da0 con 0x7fba70153190 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- 172.21.15.136:0/1585669664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba70153190 msgr2=0x7fba700c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba70153190 0x7fba700c9030 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba60009870 tx=0x7fba6001ed10 comp rx=0 tx=0).stop 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- 172.21.15.136:0/1585669664 shutdown_connections 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba700c9570 0x7fba70158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 --2- 172.21.15.136:0/1585669664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba70153190 0x7fba700c9030 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- 172.21.15.136:0/1585669664 >> 172.21.15.136:0/1585669664 conn(0x7fba700c9d50 msgr2=0x7fba700ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:31.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- 172.21.15.136:0/1585669664 shutdown_connections 2024-04-25T11:32:31.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- 172.21.15.136:0/1585669664 wait complete. 2024-04-25T11:32:31.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 Processor -- start 2024-04-25T11:32:31.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.261+0000 7fba74904640 1 -- start start 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba70153190 0x7fba700cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba701592e0 0x7fba70146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fba700c7340 con 0x7fba700c9570 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fba700c7060 con 0x7fba70153190 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba700ca870 con 0x7fba701592e0 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37170/0 (socket says 172.21.15.136:37170) 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 -- 172.21.15.136:0/1587550669 learned_addr learned my addr 172.21.15.136:0/1587550669 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba67fff640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba70153190 0x7fba700cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6ed9d640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba701592e0 0x7fba70146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 -- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba701592e0 msgr2=0x7fba70146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba701592e0 0x7fba70146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 -- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba70153190 msgr2=0x7fba700cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba70153190 0x7fba700cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 -- 172.21.15.136:0/1587550669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7015c030 con 0x7fba700c9570 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba6e59c640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fba6001d3c0 tx=0x7fba6001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600b5070 con 0x7fba700c9570 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba600b0a90 con 0x7fba700c9570 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7015b050 con 0x7fba700c9570 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba600bb410 con 0x7fba700c9570 2024-04-25T11:32:31.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba7015b2f0 con 0x7fba700c9570 2024-04-25T11:32:31.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7006a350 con 0x7fba700c9570 2024-04-25T11:32:31.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fba600c2060 con 0x7fba700c9570 2024-04-25T11:32:31.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 0x7fba50071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:31.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.265+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fba6012c930 con 0x7fba700c9570 2024-04-25T11:32:31.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.269+0000 7fba67fff640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 0x7fba50071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:31.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.269+0000 7fba67fff640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 0x7fba50071940 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fba701528b0 tx=0x7fba58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:31.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.273+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba60018870 con 0x7fba700c9570 2024-04-25T11:32:31.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.461+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba70165460 con 0x7fba700c9570 2024-04-25T11:32:31.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.461+0000 7fba66ffd640 1 -- 172.21.15.136:0/1587550669 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7fba600f9390 con 0x7fba700c9570 2024-04-25T11:32:31.452 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:31.452 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:31.452 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 msgr2=0x7fba50071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 0x7fba50071940 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fba701528b0 tx=0x7fba58008040 comp rx=0 tx=0).stop 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 msgr2=0x7fba7015baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fba6001d3c0 tx=0x7fba6001bab0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 shutdown_connections 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fba5006f430 0x7fba50071940 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba701592e0 0x7fba70146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba70153190 0x7fba700cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 --2- 172.21.15.136:0/1587550669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba700c9570 0x7fba7015baf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 >> 172.21.15.136:0/1587550669 conn(0x7fba700c9d50 msgr2=0x7fba701456b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 shutdown_connections 2024-04-25T11:32:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:31.465+0000 7fba74904640 1 -- 172.21.15.136:0/1587550669 wait complete. 2024-04-25T11:32:31.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:33.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:33.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 Processor -- start 2024-04-25T11:32:33.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 -- start start 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f45140c4750 0x7f4514152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f45141589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f45140736a0 0x7f4514073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f45140710d0 con 0x7f45140c4750 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f45140c95f0 con 0x7f45140736a0 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f451b158640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4514071300 con 0x7f4514158590 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f45140c4750 0x7f4514152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f45140c4750 0x7f4514152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37174/0 (socket says 172.21.15.136:37174) 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 -- 172.21.15.136:0/2870498038 learned_addr learned my addr 172.21.15.136:0/2870498038 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4513fff640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f45141589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f45196f5640 1 --2- 172.21.15.136:0/2870498038 >> v2:172.21.15.157:3300/0 conn(0x7f45140736a0 0x7f4514073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 -- 172.21.15.136:0/2870498038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 msgr2=0x7f45141589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:33.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f45141589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 -- 172.21.15.136:0/2870498038 >> v2:172.21.15.157:3300/0 conn(0x7f45140736a0 msgr2=0x7f4514073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:33.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 --2- 172.21.15.136:0/2870498038 >> v2:172.21.15.157:3300/0 conn(0x7f45140736a0 0x7f4514073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.645+0000 7f4518ef4640 1 -- 172.21.15.136:0/2870498038 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45140c7300 con 0x7f45140c4750 2024-04-25T11:32:33.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.649+0000 7f4518ef4640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140c4750 0x7f4514152900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f45140720d0 tx=0x7f450401d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953bbf88b6861b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.649+0000 7f45137fe640 1 -- 172.21.15.136:0/2870498038 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450400b040 con 0x7f45140c4750 2024-04-25T11:32:33.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.649+0000 7f45137fe640 1 -- 172.21.15.136:0/2870498038 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45040a9650 con 0x7f45140c4750 2024-04-25T11:32:33.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.649+0000 7f45137fe640 1 -- 172.21.15.136:0/2870498038 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45040a9a20 con 0x7f45140c4750 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- 172.21.15.136:0/2870498038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140c4750 msgr2=0x7f4514152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140c4750 0x7f4514152900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f45140720d0 tx=0x7f450401d7d0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- 172.21.15.136:0/2870498038 shutdown_connections 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- 172.21.15.136:0/2870498038 >> v2:172.21.15.157:3300/0 conn(0x7f45140736a0 0x7f4514073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f45141589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- 172.21.15.136:0/2870498038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140c4750 0x7f4514152900 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- 172.21.15.136:0/2870498038 >> 172.21.15.136:0/2870498038 conn(0x7f45140c9d50 msgr2=0x7f45140ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- 172.21.15.136:0/2870498038 shutdown_connections 2024-04-25T11:32:33.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- 172.21.15.136:0/2870498038 wait complete. 2024-04-25T11:32:33.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 Processor -- start 2024-04-25T11:32:33.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- start start 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45140c4750 0x7f45141501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f4514145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f45140ca9f0 con 0x7f45140736a0 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f45140ca870 con 0x7f45140c4750 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f451b158640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f45140c5720 con 0x7f4514158590 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f4518ef4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f4518ef4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37176/0 (socket says 172.21.15.136:37176) 2024-04-25T11:32:33.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.653+0000 7f4518ef4640 1 -- 172.21.15.136:0/3808595706 learned_addr learned my addr 172.21.15.136:0/3808595706 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45196f5640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f4514145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4513fff640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45140c4750 0x7f45141501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 -- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 msgr2=0x7f4514145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f4514145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 -- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45140c4750 msgr2=0x7f45141501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45140c4750 0x7f45141501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 -- 172.21.15.136:0/3808595706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45140cb050 con 0x7f45140736a0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45196f5640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f4514145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4518ef4640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f450401be80 tx=0x7f450401a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45040b1610 con 0x7f45140736a0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45040b17d0 con 0x7f45140736a0 2024-04-25T11:32:33.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4514152440 con 0x7f45140736a0 2024-04-25T11:32:33.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45040b1bc0 con 0x7f45140736a0 2024-04-25T11:32:33.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4514150e60 con 0x7f45140736a0 2024-04-25T11:32:33.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45140cb220 con 0x7f45140736a0 2024-04-25T11:32:33.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f45040a7040 con 0x7f45140736a0 2024-04-25T11:32:33.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 0x7f4500071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:33.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7f450412b210 con 0x7f45140736a0 2024-04-25T11:32:33.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.657+0000 7f4513fff640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 0x7f4500071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:33.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.661+0000 7f4513fff640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 0x7f4500071910 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f451413d030 tx=0x7f4508008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:33.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.661+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45040ad030 con 0x7f45140736a0 2024-04-25T11:32:33.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.861+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f451406a350 con 0x7f45140736a0 2024-04-25T11:32:33.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.861+0000 7f45117fa640 1 -- 172.21.15.136:0/3808595706 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7f450400a030 con 0x7f45140736a0 2024-04-25T11:32:33.850 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:33.850 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:33.851 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:33.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 msgr2=0x7f4500071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 0x7f4500071910 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f451413d030 tx=0x7f4508008040 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 msgr2=0x7f4514151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f450401be80 tx=0x7f450401a570 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 shutdown_connections 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f450006f400 0x7f4500071910 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4514158590 0x7f4514145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45140c4750 0x7f45141501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 --2- 172.21.15.136:0/3808595706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45140736a0 0x7f4514151f00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 >> 172.21.15.136:0/3808595706 conn(0x7f45140c9d50 msgr2=0x7f45141577d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 shutdown_connections 2024-04-25T11:32:33.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:33.865+0000 7f451b158640 1 -- 172.21.15.136:0/3808595706 wait complete. 2024-04-25T11:32:33.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:35.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:36.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 Processor -- start 2024-04-25T11:32:36.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 -- start start 2024-04-25T11:32:36.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 0x7fdca00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca00c3880 con 0x7fdca0154060 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdca00c8fd0 con 0x7fdca015aa00 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca6e3d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdca006d9b0 con 0x7fdca015a5f0 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50920/0 (socket says 172.21.15.136:50920) 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdc97fff640 1 -- 172.21.15.136:0/1211660651 learned_addr learned my addr 172.21.15.136:0/1211660651 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca53da640 1 --2- 172.21.15.136:0/1211660651 >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 0x7fdca00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.049+0000 7fdca4bd9640 1 --1- 172.21.15.136:0/1211660651 >> v1:172.21.15.157:6789/0 conn(0x7fdca0154060 0x7fdca00c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:38108/0 (socket says 172.21.15.136:38108) 2024-04-25T11:32:36.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc97fff640 1 -- 172.21.15.136:0/1211660651 >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 msgr2=0x7fdca00c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:36.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc97fff640 1 --2- 172.21.15.136:0/1211660651 >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 0x7fdca00c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc97fff640 1 -- 172.21.15.136:0/1211660651 >> v1:172.21.15.157:6789/0 conn(0x7fdca0154060 legacy=0x7fdca00c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc97fff640 1 -- 172.21.15.136:0/1211660651 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca0154470 con 0x7fdca015a5f0 2024-04-25T11:32:36.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdca53da640 1 --2- 172.21.15.136:0/1211660651 >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 0x7fdca00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:32:36.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc97fff640 1 --2- 172.21.15.136:0/1211660651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdca0154640 tx=0x7fdc9001ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=992c1e325a36fac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:36.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.053+0000 7fdc977fe640 1 -- 172.21.15.136:0/1211660651 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900b5070 con 0x7fdca015a5f0 2024-04-25T11:32:36.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdc977fe640 1 -- 172.21.15.136:0/1211660651 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc900a8a30 con 0x7fdca015a5f0 2024-04-25T11:32:36.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdc977fe640 1 -- 172.21.15.136:0/1211660651 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900a8e00 con 0x7fdca015a5f0 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- 172.21.15.136:0/1211660651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 msgr2=0x7fdca00c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/1211660651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdca0154640 tx=0x7fdc9001ed30 comp rx=0 tx=0).stop 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- 172.21.15.136:0/1211660651 shutdown_connections 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/1211660651 >> v2:172.21.15.157:3300/0 conn(0x7fdca015aa00 0x7fdca00c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/1211660651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca015a5f0 0x7fdca00c71b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- 172.21.15.136:0/1211660651 >> 172.21.15.136:0/1211660651 conn(0x7fdca00ca5a0 msgr2=0x7fdca00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- 172.21.15.136:0/1211660651 shutdown_connections 2024-04-25T11:32:36.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- 172.21.15.136:0/1211660651 wait complete. 2024-04-25T11:32:36.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 Processor -- start 2024-04-25T11:32:36.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.057+0000 7fdca6e3d640 1 -- start start 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca0154060 0x7fdca01f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 0x7fdca0074360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdca015a0d0 con 0x7fdca015a5f0 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdca006df30 con 0x7fdca015aa00 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdca0159d80 con 0x7fdca0154060 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35180/0 (socket says 172.21.15.136:35180) 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 -- 172.21.15.136:0/120831990 learned_addr learned my addr 172.21.15.136:0/120831990 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca4bd9640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca0154060 0x7fdca01f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca53da640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 0x7fdca0074360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 -- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca0154060 msgr2=0x7fdca01f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca0154060 0x7fdca01f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 -- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 msgr2=0x7fdca0074360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 0x7fdca0074360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 -- 172.21.15.136:0/120831990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca014bcf0 con 0x7fdca015a5f0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca53da640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 0x7fdca0074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc97fff640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdc9001c960 tx=0x7fdc9001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900b5070 con 0x7fdca015a5f0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc900b0a50 con 0x7fdca015a5f0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca01e53c0 con 0x7fdca015a5f0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900b0e40 con 0x7fdca015a5f0 2024-04-25T11:32:36.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca015af40 con 0x7fdca015a5f0 2024-04-25T11:32:36.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca006a350 con 0x7fdca015a5f0 2024-04-25T11:32:36.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc900b9070 con 0x7fdca015a5f0 2024-04-25T11:32:36.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 0x7fdc8c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:36.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.061+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fdc9012c7e0 con 0x7fdca015a5f0 2024-04-25T11:32:36.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.065+0000 7fdca4bd9640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 0x7fdc8c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:36.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.065+0000 7fdca4bd9640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 0x7fdc8c071940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000f70 tx=0x7fdc9c007c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:36.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.069+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdc900c1060 con 0x7fdca015a5f0 2024-04-25T11:32:36.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdca01584a0 con 0x7fdca015a5f0 2024-04-25T11:32:36.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdc957fa640 1 -- 172.21.15.136:0/120831990 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7fdc90018890 con 0x7fdca015a5f0 2024-04-25T11:32:36.245 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:36.245 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:36.246 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 msgr2=0x7fdc8c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 0x7fdc8c071940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000f70 tx=0x7fdc9c007c50 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 msgr2=0x7fdca014b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdc9001c960 tx=0x7fdc9001bad0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 shutdown_connections 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdca015aa00 0x7fdca0074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdca015a5f0 0x7fdca014b7b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdc8c06f430 0x7fdc8c071940 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 --2- 172.21.15.136:0/120831990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdca0154060 0x7fdca01f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 >> 172.21.15.136:0/120831990 conn(0x7fdca00ca5a0 msgr2=0x7fdca014d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 shutdown_connections 2024-04-25T11:32:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:36.261+0000 7fdca6e3d640 1 -- 172.21.15.136:0/120831990 wait complete. 2024-04-25T11:32:36.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:38.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:38.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 Processor -- start 2024-04-25T11:32:38.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 -- start start 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070870 0x7fa8200c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa820070c80 0x7fa8200c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa820143580 con 0x7fa820143170 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa82015dd20 con 0x7fa820070c80 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa824f63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa82015e000 con 0x7fa820070870 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa820143170 0x7fa82015d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53374/0 (socket says 172.21.15.136:53374) 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81e59c640 1 -- 172.21.15.136:0/2943817238 learned_addr learned my addr 172.21.15.136:0/2943817238 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/2943817238 >> v2:172.21.15.136:3300/0 conn(0x7fa820070c80 0x7fa8200c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81d59a640 1 -- 172.21.15.136:0/2943817238 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521435718 0 0) 0x7fa810001120 con 0x7fa820143170 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81d59a640 1 -- 172.21.15.136:0/2943817238 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa808003530 con 0x7fa820143170 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81dd9b640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070870 0x7fa8200c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81ed9d640 1 -- 172.21.15.136:0/2943817238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070870 msgr2=0x7fa8200c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:38.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070870 0x7fa8200c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81ed9d640 1 -- 172.21.15.136:0/2943817238 >> v1:172.21.15.136:6789/0 conn(0x7fa820143170 legacy=0x7fa82015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.445+0000 7fa81ed9d640 1 -- 172.21.15.136:0/2943817238 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82006fd90 con 0x7fa820070c80 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820070c80 0x7fa8200c6ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa818009d90 tx=0x7fa81801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9959c2934536ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa81d59a640 1 -- 172.21.15.136:0/2943817238 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b4070 con 0x7fa820070c80 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa81d59a640 1 -- 172.21.15.136:0/2943817238 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8180af600 con 0x7fa820070c80 2024-04-25T11:32:38.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa81d59a640 1 -- 172.21.15.136:0/2943817238 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180af9d0 con 0x7fa820070c80 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- 172.21.15.136:0/2943817238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820070c80 msgr2=0x7fa8200c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820070c80 0x7fa8200c6ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa818009d90 tx=0x7fa81801f150 comp rx=0 tx=0).stop 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- 172.21.15.136:0/2943817238 shutdown_connections 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820070c80 0x7fa8200c6ea0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 --2- 172.21.15.136:0/2943817238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070870 0x7fa8200c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- 172.21.15.136:0/2943817238 >> 172.21.15.136:0/2943817238 conn(0x7fa8200c9d50 msgr2=0x7fa8200ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- 172.21.15.136:0/2943817238 shutdown_connections 2024-04-25T11:32:38.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- 172.21.15.136:0/2943817238 wait complete. 2024-04-25T11:32:38.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 Processor -- start 2024-04-25T11:32:38.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.449+0000 7fa824f63640 1 -- start start 2024-04-25T11:32:38.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 0x7fa8201571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070c80 0x7fa8201e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa82015e2e0 con 0x7fa820143170 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa820070350 con 0x7fa820070870 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8200712a0 con 0x7fa820070c80 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35200/0 (socket says 172.21.15.136:35200) 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 -- 172.21.15.136:0/3167844723 learned_addr learned my addr 172.21.15.136:0/3167844723 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81e59c640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 0x7fa8201571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81dd9b640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070c80 0x7fa8201e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 -- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070c80 msgr2=0x7fa8201e6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070c80 0x7fa8201e6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 -- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 msgr2=0x7fa8201571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 0x7fa8201571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 -- 172.21.15.136:0/3167844723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa820155310 con 0x7fa820143170 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81e59c640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 0x7fa8201571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa81ed9d640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b2040 tx=0x7fa81801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b4040 con 0x7fa820143170 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8180c1070 con 0x7fa820143170 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8201f3a80 con 0x7fa820143170 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818009450 con 0x7fa820143170 2024-04-25T11:32:38.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa820157730 con 0x7fa820143170 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.453+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82006a350 con 0x7fa820143170 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.457+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa8180ba6b0 con 0x7fa820143170 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.457+0000 7fa80effd640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 0x7fa800071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.457+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 7165+0+0 (secure 0 0 0) 0x7fa81812ec30 con 0x7fa820143170 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.457+0000 7fa81e59c640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 0x7fa800071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:38.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.457+0000 7fa81e59c640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 0x7fa800071910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa810005890 tx=0x7fa810009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:38.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.461+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa818018cb0 con 0x7fa820143170 2024-04-25T11:32:38.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.649+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8201550a0 con 0x7fa820143170 2024-04-25T11:32:38.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.649+0000 7fa80effd640 1 -- 172.21.15.136:0/3167844723 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6929 (secure 0 0 0) 0x7fa8180fb5e0 con 0x7fa820143170 2024-04-25T11:32:38.635 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:38.636 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:31:10.115387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":6,"addr":"172.21.15.157:6839/1671830331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":1671830331},{"type":"v1","addr":"172.21.15.157:6839","nonce":1671830331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4303,"qdb_cluster":[5385,5238,5517,5036,4303]},"id":1}]} 2024-04-25T11:32:38.636 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 49 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 msgr2=0x7fa800071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 0x7fa800071910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa810005890 tx=0x7fa810009040 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 msgr2=0x7fa82015a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b2040 tx=0x7fa81801bef0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 shutdown_connections 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa820143170 0x7fa82015a1b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa80006f400 0x7fa800071910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa820070c80 0x7fa8201e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 --2- 172.21.15.136:0/3167844723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa820070870 0x7fa8201571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:38.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 >> 172.21.15.136:0/3167844723 conn(0x7fa8200c9d50 msgr2=0x7fa8200cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:38.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 shutdown_connections 2024-04-25T11:32:38.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:38.653+0000 7fa824f63640 1 -- 172.21.15.136:0/3167844723 wait complete. 2024-04-25T11:32:38.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:32:40.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.829+0000 7fc7a3698640 1 Processor -- start 2024-04-25T11:32:40.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.829+0000 7fc7a3698640 1 -- start start 2024-04-25T11:32:40.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a3698640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc79c0c4750 0x7fc79c152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a3698640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc79c158590 0x7fc79c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a3698640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c071300 con 0x7fc79c15bb80 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a3698640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc79c0710d0 con 0x7fc79c0c4750 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a3698640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc79c0c95f0 con 0x7fc79c158590 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc79c0c4750 0x7fc79c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc79c0c4750 0x7fc79c152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35216/0 (socket says 172.21.15.136:35216) 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 -- 172.21.15.136:0/3247095902 learned_addr learned my addr 172.21.15.136:0/3247095902 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a0c33640 1 --2- 172.21.15.136:0/3247095902 >> v2:172.21.15.157:3300/0 conn(0x7fc79c158590 0x7fc79c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 -- 172.21.15.136:0/3247095902 >> v2:172.21.15.157:3300/0 conn(0x7fc79c158590 msgr2=0x7fc79c1589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 --2- 172.21.15.136:0/3247095902 >> v2:172.21.15.157:3300/0 conn(0x7fc79c158590 0x7fc79c1589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 -- 172.21.15.136:0/3247095902 >> v1:172.21.15.157:6789/0 conn(0x7fc79c15bb80 legacy=0x7fc79c15bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.833+0000 7fc7a1434640 1 -- 172.21.15.136:0/3247095902 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c0c7300 con 0x7fc79c0c4750 2024-04-25T11:32:40.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a1434640 1 --2- 172.21.15.136:0/3247095902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c152900 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc79c071bd0 tx=0x7fc79801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d998d68e9bfca207 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:40.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc793fff640 1 -- 172.21.15.136:0/3247095902 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980b0070 con 0x7fc79c0c4750 2024-04-25T11:32:40.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc793fff640 1 -- 172.21.15.136:0/3247095902 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7980ab600 con 0x7fc79c0c4750 2024-04-25T11:32:40.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc793fff640 1 -- 172.21.15.136:0/3247095902 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980ab9d0 con 0x7fc79c0c4750 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- 172.21.15.136:0/3247095902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 msgr2=0x7fc79c152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 --2- 172.21.15.136:0/3247095902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c152900 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc79c071bd0 tx=0x7fc79801c400 comp rx=0 tx=0).stop 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- 172.21.15.136:0/3247095902 shutdown_connections 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 --2- 172.21.15.136:0/3247095902 >> v2:172.21.15.157:3300/0 conn(0x7fc79c158590 0x7fc79c1589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 --2- 172.21.15.136:0/3247095902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c152900 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- 172.21.15.136:0/3247095902 >> 172.21.15.136:0/3247095902 conn(0x7fc79c0c9d50 msgr2=0x7fc79c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- 172.21.15.136:0/3247095902 shutdown_connections 2024-04-25T11:32:40.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- 172.21.15.136:0/3247095902 wait complete. 2024-04-25T11:32:40.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 Processor -- start 2024-04-25T11:32:40.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.837+0000 7fc7a3698640 1 -- start start 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 0x7fc79c156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 0x7fc79c157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c073f50 con 0x7fc79c0c4750 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c073dd0 con 0x7fc79c158590 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c0ca870 con 0x7fc79c15bb80 2024-04-25T11:32:40.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35222/0 (socket says 172.21.15.136:35222) 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 -- 172.21.15.136:0/1921573102 learned_addr learned my addr 172.21.15.136:0/1921573102 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1c35640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 0x7fc79c157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a0c33640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 0x7fc79c156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 -- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 msgr2=0x7fc79c157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 0x7fc79c157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 -- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 msgr2=0x7fc79c156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 0x7fc79c156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 -- 172.21.15.136:0/1921573102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c1569d0 con 0x7fc79c0c4750 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1c35640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 0x7fc79c157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a0c33640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 0x7fc79c156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a1434640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc7980ae040 tx=0x7fc7980191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980b8020 con 0x7fc79c0c4750 2024-04-25T11:32:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7980b4410 con 0x7fc79c0c4750 2024-04-25T11:32:40.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c143670 con 0x7fc79c0c4750 2024-04-25T11:32:40.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980b0040 con 0x7fc79c0c4750 2024-04-25T11:32:40.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c0cb510 con 0x7fc79c0c4750 2024-04-25T11:32:40.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c06a350 con 0x7fc79c0c4750 2024-04-25T11:32:40.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.841+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc7980bb020 con 0x7fc79c0c4750 2024-04-25T11:32:40.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.845+0000 7fc791ffb640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 0x7fc780071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:40.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.845+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7fc798129110 con 0x7fc79c0c4750 2024-04-25T11:32:40.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.845+0000 7fc7a0c33640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 0x7fc780071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:40.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.845+0000 7fc7a0c33640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 0x7fc780071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc79c143560 tx=0x7fc78c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:40.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:40.849+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc7980c2070 con 0x7fc79c0c4750 2024-04-25T11:32:41.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.045+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc79c149420 con 0x7fc79c0c4750 2024-04-25T11:32:41.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.045+0000 7fc791ffb640 1 -- 172.21.15.136:0/1921573102 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6098 (secure 0 0 0) 0x7fc798016110 con 0x7fc79c0c4750 2024-04-25T11:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:38.913413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:32:41.033 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 50 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 msgr2=0x7fc780071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 0x7fc780071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc79c143560 tx=0x7fc78c008040 comp rx=0 tx=0).stop 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 msgr2=0x7fc79c14b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc7980ae040 tx=0x7fc7980191a0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 shutdown_connections 2024-04-25T11:32:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc78006f430 0x7fc780071940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc79c15bb80 0x7fc79c157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc79c158590 0x7fc79c156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 --2- 172.21.15.136:0/1921573102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc79c0c4750 0x7fc79c14b690 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 >> 172.21.15.136:0/1921573102 conn(0x7fc79c0c9d50 msgr2=0x7fc79c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 shutdown_connections 2024-04-25T11:32:41.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.049+0000 7fc7a3698640 1 -- 172.21.15.136:0/1921573102 wait complete. 2024-04-25T11:32:41.056 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-25T11:32:41.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:32:41.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:41.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.209+0000 7f91ea3bc640 1 Processor -- start 2024-04-25T11:32:41.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.209+0000 7f91ea3bc640 1 -- start start 2024-04-25T11:32:41.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91ea3bc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91e40cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91ea3bc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f91e4160370 0x7f91e4166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91ea3bc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91e415c6f0 con 0x7f91e415ff60 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91ea3bc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f91e40c6310 con 0x7f91e4160370 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91ea3bc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f91e415c990 con 0x7f91e415c0d0 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f91e415ff60 0x7f91e40ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:53388/0 (socket says 172.21.15.136:53388) 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 -- 172.21.15.136:0/2965053795 learned_addr learned my addr 172.21.15.136:0/2965053795 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114481948 0 0) 0x7f91e415c6f0 con 0x7f91e415ff60 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e8959640 1 --2- 172.21.15.136:0/2965053795 >> v2:172.21.15.157:3300/0 conn(0x7f91e4160370 0x7f91e4166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e3fff640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91e40cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91d0003530 con 0x7f91e415ff60 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 496703699 0 0) 0x7f91d0003530 con 0x7f91e415ff60 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 msgr2=0x7f91e40cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91e40cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 >> v2:172.21.15.157:3300/0 conn(0x7f91e4160370 msgr2=0x7f91e4166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 --2- 172.21.15.136:0/2965053795 >> v2:172.21.15.157:3300/0 conn(0x7f91e4160370 0x7f91e4166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e415cc30 con 0x7f91e415ff60 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f91cc001680 con 0x7f91e415ff60 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e8959640 1 --2- 172.21.15.136:0/2965053795 >> v2:172.21.15.157:3300/0 conn(0x7f91e4160370 0x7f91e4166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 >> v1:172.21.15.136:6789/0 conn(0x7f91e415ff60 legacy=0x7f91e40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91d00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91d0003ff0 0x7f91d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91d0004d50 0x7f91d0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f91e40c6310 con 0x7f91d0003ff0 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f91e415c990 con 0x7f91d0004d50 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f91d000bc10 con 0x7f91e415c0d0 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91d0003ff0 0x7f91d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e3fff640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91d00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e8959640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91d0004d50 0x7f91d0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 -- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 msgr2=0x7f91d00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:41.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e415c0d0 0x7f91d00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 -- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91d0004d50 msgr2=0x7f91d0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91d0004d50 0x7f91d0005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.213+0000 7f91e37fe640 1 -- 172.21.15.136:0/2965053795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d0003530 con 0x7f91d0003ff0 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91d0003ff0 0x7f91d0004810 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f91cc001060 tx=0x7f91cc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f71ca125170ace server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc007d80 con 0x7f91d0003ff0 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91cc0a9020 con 0x7f91d0003ff0 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91e2ffd640 1 -- 172.21.15.136:0/2965053795 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc0a5b40 con 0x7f91d0003ff0 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2965053795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91d0003ff0 msgr2=0x7f91d0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2965053795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91d0003ff0 0x7f91d0004810 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f91cc001060 tx=0x7f91cc01cc40 comp rx=0 tx=0).stop 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91e8959640 1 -- 172.21.15.136:0/2965053795 reap_dead start 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2965053795 shutdown_connections 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2965053795 >> 172.21.15.136:0/2965053795 conn(0x7f91e406f9d0 msgr2=0x7f91e406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2965053795 shutdown_connections 2024-04-25T11:32:41.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.217+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2965053795 wait complete. 2024-04-25T11:32:41.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 Processor -- start 2024-04-25T11:32:41.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- start start 2024-04-25T11:32:41.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91d0003ff0 0x7f91e41f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 0x7f91e40cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f91e415cf10 con 0x7f91e41e56b0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f91e415fa40 con 0x7f91e4154ef0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f91e40c9170 con 0x7f91d0003ff0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35240/0 (socket says 172.21.15.136:35240) 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 -- 172.21.15.136:0/2842387556 learned_addr learned my addr 172.21.15.136:0/2842387556 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e3fff640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91d0003ff0 0x7f91e41f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 0x7f91e40cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 -- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91d0003ff0 msgr2=0x7f91e41f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91d0003ff0 0x7f91e41f2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 -- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 msgr2=0x7f91e40cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 0x7f91e40cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 -- 172.21.15.136:0/2842387556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e4155440 con 0x7f91e41e56b0 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e37fe640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 0x7f91e40cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:41.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e8959640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f91e406f980 tx=0x7f91dc010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc00f040 con 0x7f91e41e56b0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc01b070 con 0x7f91e41e56b0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41556e0 con 0x7f91e41e56b0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc015ac0 con 0x7f91e41e56b0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e40cc610 con 0x7f91e41e56b0 2024-04-25T11:32:41.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.221+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e406a350 con 0x7f91e41e56b0 2024-04-25T11:32:41.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.225+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f91dc015c60 con 0x7f91e41e56b0 2024-04-25T11:32:41.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.225+0000 7f91e0ff9640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 0x7f91c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:41.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.225+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f91dc08f8d0 con 0x7f91e41e56b0 2024-04-25T11:32:41.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.225+0000 7f91e3fff640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 0x7f91c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:41.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.225+0000 7f91e3fff640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 0x7f91c4071910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f91d4004330 tx=0x7f91d400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:41.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.229+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f91dc05c240 con 0x7f91e41e56b0 2024-04-25T11:32:41.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91e4150310 con 0x7f91e41e56b0 2024-04-25T11:32:41.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91e0ff9640 1 -- 172.21.15.136:0/2842387556 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6098 (secure 0 0 0) 0x7f91dc05bb40 con 0x7f91e41e56b0 2024-04-25T11:32:41.410 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:41.410 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:38.913413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:32:41.410 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 50 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 msgr2=0x7f91c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 0x7f91c4071910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f91d4004330 tx=0x7f91d400b040 comp rx=0 tx=0).stop 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 msgr2=0x7f91e4149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f91e406f980 tx=0x7f91dc010900 comp rx=0 tx=0).stop 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 shutdown_connections 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91e41e56b0 0x7f91e4149b30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e4154ef0 0x7f91e40cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f91c406f400 0x7f91c4071910 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 --2- 172.21.15.136:0/2842387556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91d0003ff0 0x7f91e41f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:41.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.425+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 >> 172.21.15.136:0/2842387556 conn(0x7f91e406f9d0 msgr2=0x7f91e40c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:41.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.429+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 shutdown_connections 2024-04-25T11:32:41.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:41.429+0000 7f91ea3bc640 1 -- 172.21.15.136:0/2842387556 wait complete. 2024-04-25T11:32:41.434 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5036": { "addr": "172.21.15.157:6837/3688751078", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3688751078, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3688751078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5036, "incarnation": 44, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 13 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:32:38.913413+0000", "qdb_cluster": [ 5036, 5517, 5238, 5385 ], "qdb_leader": 5036, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5036, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:32:43.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:43.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 Processor -- start 2024-04-25T11:32:43.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 -- start start 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f09a00c79e0 0x7f09a00c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f09a015a3d0 0x7f09a015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a00c9af0 con 0x7f09a00c96e0 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f09a006cc10 con 0x7f09a015a3d0 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a7576640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f09a00cb220 con 0x7f09a00c79e0 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f09a015a3d0 0x7f09a015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f09a015a3d0 0x7f09a015a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35252/0 (socket says 172.21.15.136:35252) 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 -- 172.21.15.136:0/3928911941 learned_addr learned my addr 172.21.15.136:0/3928911941 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a4b11640 1 --2- 172.21.15.136:0/3928911941 >> v2:172.21.15.157:3300/0 conn(0x7f09a00c79e0 0x7f09a00c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 -- 172.21.15.136:0/3928911941 >> v2:172.21.15.157:3300/0 conn(0x7f09a00c79e0 msgr2=0x7f09a00c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 --2- 172.21.15.136:0/3928911941 >> v2:172.21.15.157:3300/0 conn(0x7f09a00c79e0 0x7f09a00c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 -- 172.21.15.136:0/3928911941 >> v1:172.21.15.157:6789/0 conn(0x7f09a00c96e0 legacy=0x7f09a01540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.613+0000 7f09a5b13640 1 -- 172.21.15.136:0/3928911941 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a00cb500 con 0x7f09a015a3d0 2024-04-25T11:32:43.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a5b13640 1 --2- 172.21.15.136:0/3928911941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a015a3d0 0x7f09a015a7e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c8990 tx=0x7f099801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9881d4a3855989bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:43.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f0997fff640 1 -- 172.21.15.136:0/3928911941 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a56c0 con 0x7f09a015a3d0 2024-04-25T11:32:43.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f0997fff640 1 -- 172.21.15.136:0/3928911941 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09980a5860 con 0x7f09a015a3d0 2024-04-25T11:32:43.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f0997fff640 1 -- 172.21.15.136:0/3928911941 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09980a5c30 con 0x7f09a015a3d0 2024-04-25T11:32:43.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 -- 172.21.15.136:0/3928911941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a015a3d0 msgr2=0x7f09a015a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 --2- 172.21.15.136:0/3928911941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a015a3d0 0x7f09a015a7e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c8990 tx=0x7f099801c9f0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 -- 172.21.15.136:0/3928911941 shutdown_connections 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 --2- 172.21.15.136:0/3928911941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a015a3d0 0x7f09a015a7e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 --2- 172.21.15.136:0/3928911941 >> v2:172.21.15.157:3300/0 conn(0x7f09a00c79e0 0x7f09a00c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 -- 172.21.15.136:0/3928911941 >> 172.21.15.136:0/3928911941 conn(0x7f09a006f9d0 msgr2=0x7f09a006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 -- 172.21.15.136:0/3928911941 shutdown_connections 2024-04-25T11:32:43.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.617+0000 7f09a7576640 1 -- 172.21.15.136:0/3928911941 wait complete. 2024-04-25T11:32:43.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 Processor -- start 2024-04-25T11:32:43.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- start start 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a00c96e0 0x7f09a0143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 0x7f09a0140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f09a00ced40 con 0x7f09a00c79e0 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f09a006d440 con 0x7f09a00c96e0 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f09a0143e80 con 0x7f09a015a3d0 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35262/0 (socket says 172.21.15.136:35262) 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 -- 172.21.15.136:0/3792415596 learned_addr learned my addr 172.21.15.136:0/3792415596 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a4b11640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a00c96e0 0x7f09a0143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5b13640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 0x7f09a0140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 -- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 msgr2=0x7f09a0140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 0x7f09a0140a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 -- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a00c96e0 msgr2=0x7f09a0143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a00c96e0 0x7f09a0143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 -- 172.21.15.136:0/3792415596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a01439b0 con 0x7f09a00c79e0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5b13640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 0x7f09a0140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a5312640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f099000c370 tx=0x7f099000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09900052d0 con 0x7f09a00c79e0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0990005470 con 0x7f09a00c79e0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a00cd890 con 0x7f09a00c79e0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990007260 con 0x7f09a00c79e0 2024-04-25T11:32:43.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a0142c70 con 0x7f09a00c79e0 2024-04-25T11:32:43.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.621+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a01428d0 con 0x7f09a00c79e0 2024-04-25T11:32:43.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.625+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0990007400 con 0x7f09a00c79e0 2024-04-25T11:32:43.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.625+0000 7f0995ffb640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 0x7f0984071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:43.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.625+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f099008ed80 con 0x7f09a00c79e0 2024-04-25T11:32:43.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.625+0000 7f09a4b11640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 0x7f0984071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:43.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.625+0000 7f09a4b11640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 0x7f0984071940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f099c009a50 tx=0x7f099c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:43.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.629+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f09a01428d0 con 0x7f09a00c79e0 2024-04-25T11:32:43.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.825+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09a006a350 con 0x7f09a00c79e0 2024-04-25T11:32:43.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.825+0000 7f0995ffb640 1 -- 172.21.15.136:0/3792415596 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6098 (secure 0 0 0) 0x7f099005b650 con 0x7f09a00c79e0 2024-04-25T11:32:43.811 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:43.811 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:38.913413+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:32:43.812 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 50 2024-04-25T11:32:43.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 msgr2=0x7f0984071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 0x7f0984071940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f099c009a50 tx=0x7f099c008040 comp rx=0 tx=0).stop 2024-04-25T11:32:43.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 msgr2=0x7f09a0141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f099000c370 tx=0x7f099000c830 comp rx=0 tx=0).stop 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 shutdown_connections 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f098406f430 0x7f0984071940 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f09a015a3d0 0x7f09a0140a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a00c96e0 0x7f09a0143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 --2- 172.21.15.136:0/3792415596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09a00c79e0 0x7f09a0141320 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 >> 172.21.15.136:0/3792415596 conn(0x7f09a006f9d0 msgr2=0x7f09a0147630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 shutdown_connections 2024-04-25T11:32:43.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:43.829+0000 7f09a7576640 1 -- 172.21.15.136:0/3792415596 wait complete. 2024-04-25T11:32:43.835 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:32:45.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:45.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 Processor -- start 2024-04-25T11:32:45.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 -- start start 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efbf8156750 0x7efbf80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbf815c410 con 0x7efbf80ca7d0 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbf815c6b0 con 0x7efbf80cfba0 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbfcda2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7efbf815c990 con 0x7efbf8156750 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efbf8156750 0x7efbf80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efbf8156750 0x7efbf80ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35426/0 (socket says 172.21.15.136:35426) 2024-04-25T11:32:45.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 -- 172.21.15.136:0/3005830031 learned_addr learned my addr 172.21.15.136:0/3005830031 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:46.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 -- 172.21.15.136:0/3005830031 >> v1:172.21.15.157:6789/0 conn(0x7efbf80cfba0 legacy=0x7efbf806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:46.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 -- 172.21.15.136:0/3005830031 >> v1:172.21.15.136:6789/0 conn(0x7efbf80ca7d0 legacy=0x7efbf80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.013+0000 7efbf659c640 1 -- 172.21.15.136:0/3005830031 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf815cc70 con 0x7efbf8156750 2024-04-25T11:32:46.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbf659c640 1 --2- 172.21.15.136:0/3005830031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf8156750 0x7efbf80ca290 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009e50 tx=0x7efbe801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fcc81a97781ec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:46.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbf559a640 1 -- 172.21.15.136:0/3005830031 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe80a56c0 con 0x7efbf8156750 2024-04-25T11:32:46.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbf559a640 1 -- 172.21.15.136:0/3005830031 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbe80a5860 con 0x7efbf8156750 2024-04-25T11:32:46.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbf559a640 1 -- 172.21.15.136:0/3005830031 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe80a5c30 con 0x7efbf8156750 2024-04-25T11:32:46.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 -- 172.21.15.136:0/3005830031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf8156750 msgr2=0x7efbf80ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3005830031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf8156750 0x7efbf80ca290 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009e50 tx=0x7efbe801c9f0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 -- 172.21.15.136:0/3005830031 shutdown_connections 2024-04-25T11:32:46.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3005830031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf8156750 0x7efbf80ca290 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 -- 172.21.15.136:0/3005830031 >> 172.21.15.136:0/3005830031 conn(0x7efbf8070f70 msgr2=0x7efbf8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:46.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 -- 172.21.15.136:0/3005830031 shutdown_connections 2024-04-25T11:32:46.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.017+0000 7efbfcda2640 1 -- 172.21.15.136:0/3005830031 wait complete. 2024-04-25T11:32:46.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 Processor -- start 2024-04-25T11:32:46.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- start start 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 0x7efbf8141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efbf8142700 0x7efbf80b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efbf816a8f0 con 0x7efbf80cfba0 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efbf806da70 con 0x7efbf80ca7d0 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efbf816aa70 con 0x7efbf8142700 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:46.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35434/0 (socket says 172.21.15.136:35434) 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 -- 172.21.15.136:0/3705704598 learned_addr learned my addr 172.21.15.136:0/3705704598 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf659c640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 0x7efbf8141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf6d9d640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efbf8142700 0x7efbf80b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 -- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efbf8142700 msgr2=0x7efbf80b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efbf8142700 0x7efbf80b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 -- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 msgr2=0x7efbf8141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 0x7efbf8141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 -- 172.21.15.136:0/3705704598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf81f9930 con 0x7efbf80cfba0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf659c640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 0x7efbf8141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbf5d9b640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c340 tx=0x7efbec00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec0052d0 con 0x7efbf80cfba0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbec005470 con 0x7efbf80cfba0 2024-04-25T11:32:46.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf81f9bd0 con 0x7efbf80cfba0 2024-04-25T11:32:46.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81490c0 con 0x7efbf80cfba0 2024-04-25T11:32:46.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec007260 con 0x7efbf80cfba0 2024-04-25T11:32:46.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.021+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbdc0052a0 con 0x7efbf80cfba0 2024-04-25T11:32:46.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.025+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efbec007400 con 0x7efbf80cfba0 2024-04-25T11:32:46.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.025+0000 7efbe6ffd640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 0x7efbd8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:46.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.025+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7efbec08eca0 con 0x7efbf80cfba0 2024-04-25T11:32:46.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.025+0000 7efbf659c640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 0x7efbd8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:46.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.025+0000 7efbf659c640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 0x7efbd8071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efbe80ae000 tx=0x7efbe80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:46.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.029+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efbec05b580 con 0x7efbf80cfba0 2024-04-25T11:32:46.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.229+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efbdc005860 con 0x7efbf80cfba0 2024-04-25T11:32:46.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.229+0000 7efbe6ffd640 1 -- 172.21.15.136:0/3705704598 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6100 (secure 0 0 0) 0x7efbec05ae80 con 0x7efbf80cfba0 2024-04-25T11:32:46.215 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:46.215 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:44.252963+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:resolve","state_seq":23,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5385,5238,5517,5036]},"id":1}]} 2024-04-25T11:32:46.215 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 51 2024-04-25T11:32:46.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 msgr2=0x7efbd8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 0x7efbd8071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efbe80ae000 tx=0x7efbe80a7070 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 msgr2=0x7efbf81498c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efbec00c340 tx=0x7efbec00c800 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 shutdown_connections 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efbd806f430 0x7efbd8071940 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efbf8142700 0x7efbf80b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efbf80cfba0 0x7efbf81498c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 --2- 172.21.15.136:0/3705704598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efbf80ca7d0 0x7efbf8141400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 >> 172.21.15.136:0/3705704598 conn(0x7efbf8070f70 msgr2=0x7efbf814edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 shutdown_connections 2024-04-25T11:32:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:46.233+0000 7efbfcda2640 1 -- 172.21.15.136:0/3705704598 wait complete. 2024-04-25T11:32:46.240 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:32:48.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:48.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 Processor -- start 2024-04-25T11:32:48.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 -- start start 2024-04-25T11:32:48.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60600c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:48.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60600c8d40 con 0x7f606015aa00 2024-04-25T11:32:48.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60600c3880 con 0x7f606015a5f0 2024-04-25T11:32:48.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f6065de8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f60600c8fd0 con 0x7f6060154060 2024-04-25T11:32:48.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f605ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f606015aa00 0x7f6060164700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46672/0 (socket says 172.21.15.136:46672) 2024-04-25T11:32:48.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f605ffff640 1 -- 172.21.15.136:0/794062048 learned_addr learned my addr 172.21.15.136:0/794062048 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:48.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.417+0000 7f605f7fe640 1 --2- 172.21.15.136:0/794062048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60600c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:48.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234874647 0 0) 0x7f60600c8d40 con 0x7f606015aa00 2024-04-25T11:32:48.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f604c003530 con 0x7f606015aa00 2024-04-25T11:32:48.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3315117961 0 0) 0x7f60600c3880 con 0x7f606015a5f0 2024-04-25T11:32:48.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60600c8d40 con 0x7f606015a5f0 2024-04-25T11:32:48.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605f7fe640 1 -- 172.21.15.136:0/794062048 >> v1:172.21.15.157:6789/0 conn(0x7f606015a5f0 legacy=0x7f60600c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605f7fe640 1 -- 172.21.15.136:0/794062048 >> v1:172.21.15.136:6789/0 conn(0x7f606015aa00 legacy=0x7f6060164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605f7fe640 1 -- 172.21.15.136:0/794062048 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606006d9b0 con 0x7f6060154060 2024-04-25T11:32:48.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605f7fe640 1 --2- 172.21.15.136:0/794062048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60600c8800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60500097f0 tx=0x7f605001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8728a6d8df235a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:48.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500b5070 con 0x7f6060154060 2024-04-25T11:32:48.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60500a89d0 con 0x7f6060154060 2024-04-25T11:32:48.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f605effd640 1 -- 172.21.15.136:0/794062048 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500a8d80 con 0x7f6060154060 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 -- 172.21.15.136:0/794062048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 msgr2=0x7f60600c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 --2- 172.21.15.136:0/794062048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60600c8800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60500097f0 tx=0x7f605001ecd0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 -- 172.21.15.136:0/794062048 shutdown_connections 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 --2- 172.21.15.136:0/794062048 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60600c8800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 -- 172.21.15.136:0/794062048 >> 172.21.15.136:0/794062048 conn(0x7f60600ca5a0 msgr2=0x7f60600ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 -- 172.21.15.136:0/794062048 shutdown_connections 2024-04-25T11:32:48.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.421+0000 7f6065de8640 1 -- 172.21.15.136:0/794062048 wait complete. 2024-04-25T11:32:48.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 Processor -- start 2024-04-25T11:32:48.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- start start 2024-04-25T11:32:48.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60601e54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:48.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 0x7f606014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:48.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:48.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f606006df30 con 0x7f606015aa00 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6060154470 con 0x7f606015a5f0 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f606015a0d0 con 0x7f6060154060 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35450/0 (socket says 172.21.15.136:35450) 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 -- 172.21.15.136:0/2586772119 learned_addr learned my addr 172.21.15.136:0/2586772119 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605f7fe640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60601e54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6057fff640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 0x7f606014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 -- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 msgr2=0x7f60601e54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60601e54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 -- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 msgr2=0x7f606014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 0x7f606014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 -- 172.21.15.136:0/2586772119 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60601f45b0 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6057fff640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 0x7f606014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605ffff640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f605800cae0 tx=0x7f605800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058017070 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605800ac40 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60601f4850 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058012600 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.425+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60601f0820 con 0x7f606015aa00 2024-04-25T11:32:48.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606006a350 con 0x7f606015aa00 2024-04-25T11:32:48.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f605801a070 con 0x7f606015aa00 2024-04-25T11:32:48.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f605cff9640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:48.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f605808f430 con 0x7f606015aa00 2024-04-25T11:32:48.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f605f7fe640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:48.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.429+0000 7f605f7fe640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 0x7f6040071910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60500b2000 tx=0x7f60500aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:48.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.433+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f60601f45b0 con 0x7f606015aa00 2024-04-25T11:32:48.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.633+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60601576a0 con 0x7f606015aa00 2024-04-25T11:32:48.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.633+0000 7f605cff9640 1 -- 172.21.15.136:0/2586772119 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6100 (secure 0 0 0) 0x7f605805b7e0 con 0x7f606015aa00 2024-04-25T11:32:48.619 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:48.619 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:44.252963+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:resolve","state_seq":23,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5385,5238,5517,5036]},"id":1}]} 2024-04-25T11:32:48.619 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 51 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 msgr2=0x7f6040071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 0x7f6040071910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60500b2000 tx=0x7f60500aa070 comp rx=0 tx=0).stop 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 msgr2=0x7f60601517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f605800cae0 tx=0x7f605800a060 comp rx=0 tx=0).stop 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 shutdown_connections 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f606015aa00 0x7f60601517d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f606015a5f0 0x7f606014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 --2- 172.21.15.136:0/2586772119 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6060154060 0x7f60601e54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:48.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 >> 172.21.15.136:0/2586772119 conn(0x7f60600ca5a0 msgr2=0x7f606014d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:48.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 shutdown_connections 2024-04-25T11:32:48.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:48.637+0000 7f6065de8640 1 -- 172.21.15.136:0/2586772119 wait complete. 2024-04-25T11:32:48.641 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:32:50.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:50.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.817+0000 7f9f6f868640 1 Processor -- start 2024-04-25T11:32:50.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.817+0000 7f9f6f868640 1 -- start start 2024-04-25T11:32:50.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6f868640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9f68153190 0x7f9f680c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6f868640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f680c8090 con 0x7f9f681594b0 2024-04-25T11:32:50.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6f868640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f680c6aa0 con 0x7f9f680c9570 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6f868640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9f680c8370 con 0x7f9f68153190 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6de05640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9f681594b0 0x7f9f68163660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46680/0 (socket says 172.21.15.136:46680) 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6de05640 1 -- 172.21.15.136:0/2698004115 learned_addr learned my addr 172.21.15.136:0/2698004115 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 691049597 0 0) 0x7f9f680c8090 con 0x7f9f681594b0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f54005740 con 0x7f9f681594b0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 --2- 172.21.15.136:0/2698004115 >> v2:172.21.15.157:3300/0 conn(0x7f9f68153190 0x7f9f680c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 746989752 0 0) 0x7f9f54005740 con 0x7f9f681594b0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 >> v2:172.21.15.157:3300/0 conn(0x7f9f68153190 msgr2=0x7f9f680c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 --2- 172.21.15.136:0/2698004115 >> v2:172.21.15.157:3300/0 conn(0x7f9f68153190 0x7f9f680c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 >> v1:172.21.15.157:6789/0 conn(0x7f9f680c9570 legacy=0x7f9f68158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f680c6d80 con 0x7f9f681594b0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f9f64001680 con 0x7f9f681594b0 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 >> v1:172.21.15.136:6789/0 conn(0x7f9f681594b0 legacy=0x7f9f68163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f54006090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f54006a20 0x7f9f54006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f54007570 0x7f9f5400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680c8370 con 0x7f9f54005c20 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5400e460 con 0x7f9f54007570 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f5400e660 con 0x7f9f54006a20 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f54006090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6de05640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f54007570 0x7f9f5400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 -- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f54006a20 msgr2=0x7f9f54006e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f54006a20 0x7f9f54006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f54006a20 0x7f9f54006e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 -- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f54007570 msgr2=0x7f9f5400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f54007570 0x7f9f5400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6de05640 1 -- 172.21.15.136:0/2698004115 reap_dead start 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6d604640 1 -- 172.21.15.136:0/2698004115 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f54005740 con 0x7f9f54005c20 2024-04-25T11:32:50.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.821+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f54006a20 0x7f9f54006e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6d604640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f54006090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9f60007360 tx=0x7f9f6001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68feffacd68ce14e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f600b2070 con 0x7f9f54005c20 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f600a5d20 con 0x7f9f54005c20 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f5ffff640 1 -- 172.21.15.136:0/2698004115 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60009be0 con 0x7f9f54005c20 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 -- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 msgr2=0x7f9f54006090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f54006090 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9f60007360 tx=0x7f9f6001ce20 comp rx=0 tx=0).stop 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 -- 172.21.15.136:0/2698004115 shutdown_connections 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 --2- 172.21.15.136:0/2698004115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f54006090 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 -- 172.21.15.136:0/2698004115 >> 172.21.15.136:0/2698004115 conn(0x7f9f680c9d50 msgr2=0x7f9f680ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 -- 172.21.15.136:0/2698004115 shutdown_connections 2024-04-25T11:32:50.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.825+0000 7f9f6f868640 1 -- 172.21.15.136:0/2698004115 wait complete. 2024-04-25T11:32:50.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 Processor -- start 2024-04-25T11:32:50.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- start start 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 0x7f9f68158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 0x7f9f68147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680ca870 con 0x7f9f54005c20 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f680c7340 con 0x7f9f681e20f0 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f680ca9f0 con 0x7f9f68147a80 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35476/0 (socket says 172.21.15.136:35476) 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 -- 172.21.15.136:0/3383745310 learned_addr learned my addr 172.21.15.136:0/3383745310 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 0x7f9f68158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6de05640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 0x7f9f68147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 -- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 msgr2=0x7f9f68147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 0x7f9f68147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 -- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 msgr2=0x7f9f68158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 0x7f9f68158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 -- 172.21.15.136:0/3383745310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f681f2400 con 0x7f9f54005c20 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6de05640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 0x7f9f68147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 0x7f9f68158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6d604640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000f90 tx=0x7f9f60019bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f600b2040 con 0x7f9f54005c20 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f600adb20 con 0x7f9f54005c20 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f681483d0 con 0x7f9f54005c20 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f60009be0 con 0x7f9f54005c20 2024-04-25T11:32:50.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f680cce10 con 0x7f9f54005c20 2024-04-25T11:32:50.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.829+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f6806a350 con 0x7f9f54005c20 2024-04-25T11:32:50.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.833+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9f600039a0 con 0x7f9f54005c20 2024-04-25T11:32:50.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.833+0000 7f9f5dffb640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 0x7f9f4c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:50.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.833+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f9f60129d00 con 0x7f9f54005c20 2024-04-25T11:32:50.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.833+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 0x7f9f4c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:50.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.833+0000 7f9f6ce03640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 0x7f9f4c071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f68151f60 tx=0x7f9f58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:50.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:50.837+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f600be060 con 0x7f9f54005c20 2024-04-25T11:32:51.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f68071300 con 0x7f9f54005c20 2024-04-25T11:32:51.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f5dffb640 1 -- 172.21.15.136:0/3383745310 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6102 (secure 0 0 0) 0x7f9f681f2400 con 0x7f9f54005c20 2024-04-25T11:32:51.021 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:51.021 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:49.299020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:32:51.021 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 52 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 msgr2=0x7f9f4c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 0x7f9f4c071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9f68151f60 tx=0x7f9f58009040 comp rx=0 tx=0).stop 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 msgr2=0x7f9f681f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000f90 tx=0x7f9f60019bc0 comp rx=0 tx=0).stop 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 shutdown_connections 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f4c06f400 0x7f9f4c071910 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f68147a80 0x7f9f68147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f681e20f0 0x7f9f68158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:51.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 --2- 172.21.15.136:0/3383745310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f54005c20 0x7f9f681f2e80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:51.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.037+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 >> 172.21.15.136:0/3383745310 conn(0x7f9f680c9d50 msgr2=0x7f9f6806fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:51.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.041+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 shutdown_connections 2024-04-25T11:32:51.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:51.041+0000 7f9f6f868640 1 -- 172.21.15.136:0/3383745310 wait complete. 2024-04-25T11:32:51.047 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:32:52.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:32:52.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:32:53.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:53.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 Processor -- start 2024-04-25T11:32:53.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 -- start start 2024-04-25T11:32:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038158da0 0x7fd0381591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd038161650 0x7fd038163a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0381596f0 con 0x7fd038153190 2024-04-25T11:32:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd0380c8370 con 0x7fd038161650 2024-04-25T11:32:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03f760640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0380c8090 con 0x7fd038158da0 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03dcfd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd038161650 0x7fd038163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03ccfb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038158da0 0x7fd0381591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03d4fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd038153190 0x7fd0380c9030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:55586/0 (socket says 172.21.15.136:55586) 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03d4fc640 1 -- 172.21.15.136:0/988958157 learned_addr learned my addr 172.21.15.136:0/988958157 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.225+0000 7fd03dcfd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd038161650 0x7fd038163a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:49068/0 (socket says 172.21.15.136:49068) 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03dcfd640 1 -- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038158da0 msgr2=0x7fd0381591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03dcfd640 1 --2- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038158da0 0x7fd0381591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03dcfd640 1 -- 172.21.15.136:0/988958157 >> v1:172.21.15.157:6789/0 conn(0x7fd038153190 legacy=0x7fd0380c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03dcfd640 1 -- 172.21.15.136:0/988958157 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0380c6aa0 con 0x7fd038161650 2024-04-25T11:32:53.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03dcfd640 1 --2- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038161650 0x7fd038163a70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd030009d90 tx=0x7fd03001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18d2f4c12065348 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:53.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd02ffff640 1 -- 172.21.15.136:0/988958157 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0300b4070 con 0x7fd038161650 2024-04-25T11:32:53.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd02ffff640 1 -- 172.21.15.136:0/988958157 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0300af600 con 0x7fd038161650 2024-04-25T11:32:53.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd02ffff640 1 -- 172.21.15.136:0/988958157 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0300af9d0 con 0x7fd038161650 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 -- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038161650 msgr2=0x7fd038163a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 --2- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038161650 0x7fd038163a70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd030009d90 tx=0x7fd03001f150 comp rx=0 tx=0).stop 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 -- 172.21.15.136:0/988958157 shutdown_connections 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 --2- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038161650 0x7fd038163a70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 --2- 172.21.15.136:0/988958157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038158da0 0x7fd0381591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.229+0000 7fd03f760640 1 -- 172.21.15.136:0/988958157 >> 172.21.15.136:0/988958157 conn(0x7fd0380c9d50 msgr2=0x7fd0380ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- 172.21.15.136:0/988958157 shutdown_connections 2024-04-25T11:32:53.216 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- 172.21.15.136:0/988958157 wait complete. 2024-04-25T11:32:53.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 Processor -- start 2024-04-25T11:32:53.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- start start 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 0x7fd038156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038161650 0x7fd03814d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd0380c7340 con 0x7fd038153190 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd0380c7060 con 0x7fd038158da0 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03f760640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd0380ca870 con 0x7fd038161650 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35486/0 (socket says 172.21.15.136:35486) 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 -- 172.21.15.136:0/2640571150 learned_addr learned my addr 172.21.15.136:0/2640571150 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03ccfb640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 0x7fd038156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03dcfd640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038161650 0x7fd03814d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 -- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038161650 msgr2=0x7fd03814d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038161650 0x7fd03814d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 -- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 msgr2=0x7fd038156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 0x7fd038156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03d4fc640 1 -- 172.21.15.136:0/2640571150 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0381f2800 con 0x7fd038153190 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.233+0000 7fd03ccfb640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 0x7fd038156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03d4fc640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c370 tx=0x7fd02800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028017070 con 0x7fd038153190 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd028005cf0 con 0x7fd038153190 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0280072f0 con 0x7fd038153190 2024-04-25T11:32:53.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd038152920 con 0x7fd038153190 2024-04-25T11:32:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0381f2da0 con 0x7fd038153190 2024-04-25T11:32:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd03806a350 con 0x7fd038153190 2024-04-25T11:32:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd02801a070 con 0x7fd038153190 2024-04-25T11:32:53.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 0x7fd01c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:53.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7fd02808ee50 con 0x7fd038153190 2024-04-25T11:32:53.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03ccfb640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 0x7fd01c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:53.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.237+0000 7fd03ccfb640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 0x7fd01c071910 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd038158970 tx=0x7fd034008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:53.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.241+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd028021020 con 0x7fd038153190 2024-04-25T11:32:53.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.437+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd038071300 con 0x7fd038153190 2024-04-25T11:32:53.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd02dffb640 1 -- 172.21.15.136:0/2640571150 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6102 (secure 0 0 0) 0x7fd02805b930 con 0x7fd038153190 2024-04-25T11:32:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:53.425 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:49.299020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5036,5517,5238,5385]},"id":1}]} 2024-04-25T11:32:53.425 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 52 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 msgr2=0x7fd01c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 0x7fd01c071910 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd038158970 tx=0x7fd034008040 comp rx=0 tx=0).stop 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 msgr2=0x7fd038151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c370 tx=0x7fd02800c830 comp rx=0 tx=0).stop 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 shutdown_connections 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd01c06f400 0x7fd01c071910 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd038161650 0x7fd03814d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd038158da0 0x7fd038156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 --2- 172.21.15.136:0/2640571150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd038153190 0x7fd038151c00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:53.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 >> 172.21.15.136:0/2640571150 conn(0x7fd0380c9d50 msgr2=0x7fd038147c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:53.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 shutdown_connections 2024-04-25T11:32:53.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:53.441+0000 7fd03f760640 1 -- 172.21.15.136:0/2640571150 wait complete. 2024-04-25T11:32:53.449 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5036": { "addr": "172.21.15.157:6837/3688751078", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3688751078, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3688751078, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5036, "incarnation": 44, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 13 }, "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 25 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:32:49.299020+0000", "qdb_cluster": [ 5036, 5517, 5238, 5385 ], "qdb_leader": 5036, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5036, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:32:55.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:55.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.625+0000 7f43cdf65640 1 Processor -- start 2024-04-25T11:32:55.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.625+0000 7f43cdf65640 1 -- start start 2024-04-25T11:32:55.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43cdf65640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:55.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43cdf65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c8143580 con 0x7f43c8070870 2024-04-25T11:32:55.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43cdf65640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c815dd20 con 0x7f43c8070c80 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43cdf65640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f43c815e000 con 0x7f43c8143170 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43bffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f43c8070870 0x7f43c80c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43664/0 (socket says 172.21.15.136:43664) 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43bffff640 1 -- 172.21.15.136:0/3432202266 learned_addr learned my addr 172.21.15.136:0/3432202266 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c77fe640 1 --2- 172.21.15.136:0/3432202266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2668006322 0 0) 0x7f43b0001120 con 0x7f43c8070870 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b4003530 con 0x7f43c8070870 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3022362112 0 0) 0x7f43c815dd20 con 0x7f43c8070c80 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b0001120 con 0x7f43c8070c80 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c77fe640 1 -- 172.21.15.136:0/3432202266 >> v1:172.21.15.157:6789/0 conn(0x7f43c8070c80 legacy=0x7f43c8164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c77fe640 1 -- 172.21.15.136:0/3432202266 >> v1:172.21.15.136:6789/0 conn(0x7f43c8070870 legacy=0x7f43c80c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.629+0000 7f43c77fe640 1 -- 172.21.15.136:0/3432202266 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c806fd90 con 0x7f43c8143170 2024-04-25T11:32:55.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43c77fe640 1 --2- 172.21.15.136:0/3432202266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c815d400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3cbc3e27e26f0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:55.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b80b5070 con 0x7f43c8143170 2024-04-25T11:32:55.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b80a89d0 con 0x7f43c8143170 2024-04-25T11:32:55.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43c6ffd640 1 -- 172.21.15.136:0/3432202266 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b80a8d80 con 0x7f43c8143170 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 -- 172.21.15.136:0/3432202266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 msgr2=0x7f43c815d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 --2- 172.21.15.136:0/3432202266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c815d400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 -- 172.21.15.136:0/3432202266 shutdown_connections 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 --2- 172.21.15.136:0/3432202266 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c815d400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 -- 172.21.15.136:0/3432202266 >> 172.21.15.136:0/3432202266 conn(0x7f43c80c9d50 msgr2=0x7f43c80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 -- 172.21.15.136:0/3432202266 shutdown_connections 2024-04-25T11:32:55.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.633+0000 7f43cdf65640 1 -- 172.21.15.136:0/3432202266 wait complete. 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 Processor -- start 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- start start 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 0x7f43c8153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c80cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43c815e2e0 con 0x7f43c8070c80 2024-04-25T11:32:55.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8070350 con 0x7f43c8070870 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f43c80712a0 con 0x7f43c8143170 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43300/0 (socket says 172.21.15.136:43300) 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 -- 172.21.15.136:0/4187094866 learned_addr learned my addr 172.21.15.136:0/4187094866 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c77fe640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 0x7f43c8153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c7fff640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c80cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 -- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 msgr2=0x7f43c80cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c80cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 -- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 msgr2=0x7f43c8153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 0x7f43c8153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 -- 172.21.15.136:0/4187094866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c815d1e0 con 0x7f43c8070c80 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c77fe640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 0x7f43c8153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43bffff640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f43b000c730 tx=0x7f43b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:55.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0017070 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b0005aa0 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0007070 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c81f3480 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c813f5e0 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.637+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c806a350 con 0x7f43c8070c80 2024-04-25T11:32:55.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.641+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f43b001a070 con 0x7f43c8070c80 2024-04-25T11:32:55.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.641+0000 7f43c4ff9640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 0x7f43a8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:55.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.641+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f43b008ed70 con 0x7f43c8070c80 2024-04-25T11:32:55.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.641+0000 7f43c77fe640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 0x7f43a8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:55.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.641+0000 7f43c77fe640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 0x7f43a8071910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f43b80b2000 tx=0x7f43b80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:55.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.645+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f43c8143580 con 0x7f43c8070c80 2024-04-25T11:32:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c8142b90 con 0x7f43c8070c80 2024-04-25T11:32:55.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43c4ff9640 1 -- 172.21.15.136:0/4187094866 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6099 (secure 0 0 0) 0x7f43b005b6e0 con 0x7f43c8070c80 2024-04-25T11:32:55.826 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:55.826 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:54.704787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5385,5238,5517,5036]},"id":1}]} 2024-04-25T11:32:55.826 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 53 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 msgr2=0x7f43a8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 0x7f43a8071910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f43b80b2000 tx=0x7f43b80aa070 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 msgr2=0x7f43c80cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.841+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f43b000c730 tx=0x7f43b000cbf0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 shutdown_connections 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f43a806f400 0x7f43a8071910 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f43c8143170 0x7f43c80cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8070c80 0x7f43c80cc2b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 --2- 172.21.15.136:0/4187094866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f43c8070870 0x7f43c8153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:55.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 >> 172.21.15.136:0/4187094866 conn(0x7f43c80c9d50 msgr2=0x7f43c813e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 shutdown_connections 2024-04-25T11:32:55.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:55.845+0000 7f43cdf65640 1 -- 172.21.15.136:0/4187094866 wait complete. 2024-04-25T11:32:55.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:32:57.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:32:58.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.025+0000 7f8dbab77640 1 Processor -- start 2024-04-25T11:32:58.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.025+0000 7f8dbab77640 1 -- start start 2024-04-25T11:32:58.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8dbab77640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8db40c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8dbab77640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8db41594b0 0x7f8db4163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8dbab77640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db40c8090 con 0x7f8db40c9570 2024-04-25T11:32:58.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8dbab77640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8db40c8370 con 0x7f8db41594b0 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8dbab77640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8db40c6aa0 con 0x7f8db4153190 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8db40c9570 0x7f8db4158da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43672/0 (socket says 172.21.15.136:43672) 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db3fff640 1 -- 172.21.15.136:0/2814706468 learned_addr learned my addr 172.21.15.136:0/2814706468 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8db40c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db9114640 1 --2- 172.21.15.136:0/2814706468 >> v2:172.21.15.157:3300/0 conn(0x7f8db41594b0 0x7f8db4163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190762171 0 0) 0x7f8db40c8090 con 0x7f8db40c9570 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d9c003530 con 0x7f8db40c9570 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3666131196 0 0) 0x7f8d9c003530 con 0x7f8db40c9570 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 msgr2=0x7f8db40c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8db40c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 >> v2:172.21.15.157:3300/0 conn(0x7f8db41594b0 msgr2=0x7f8db4163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 --2- 172.21.15.136:0/2814706468 >> v2:172.21.15.157:3300/0 conn(0x7f8db41594b0 0x7f8db4163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db40c6d80 con 0x7f8db40c9570 2024-04-25T11:32:58.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8da8001680 con 0x7f8db40c9570 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 >> v1:172.21.15.136:6789/0 conn(0x7f8db40c9570 legacy=0x7f8db4158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8d9c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d9c004d50 0x7f8d9c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8db40c8370 con 0x7f8d9c0039c0 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8db40c6aa0 con 0x7f8d9c004d50 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c00ba50 con 0x7f8db4153190 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db9114640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d9c004d50 0x7f8d9c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db3fff640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8d9c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 -- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 msgr2=0x7f8d9c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db4153190 0x7f8d9c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 -- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d9c004d50 msgr2=0x7f8d9c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:32:58.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d9c004d50 0x7f8d9c005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.029+0000 7f8db8913640 1 -- 172.21.15.136:0/2814706468 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c003530 con 0x7f8d9c0039c0 2024-04-25T11:32:58.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8db8913640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003e30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009ab0 tx=0x7f8da401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59759fd30cca8e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da40b2070 con 0x7f8d9c0039c0 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da8001d40 con 0x7f8d9c0039c0 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8db37fe640 1 -- 172.21.15.136:0/2814706468 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da40096b0 con 0x7f8d9c0039c0 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 -- 172.21.15.136:0/2814706468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d9c0039c0 msgr2=0x7f8d9c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2814706468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003e30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009ab0 tx=0x7f8da401ca70 comp rx=0 tx=0).stop 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8db9114640 1 -- 172.21.15.136:0/2814706468 reap_dead start 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 -- 172.21.15.136:0/2814706468 shutdown_connections 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 -- 172.21.15.136:0/2814706468 >> 172.21.15.136:0/2814706468 conn(0x7f8db40c9d50 msgr2=0x7f8db40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 -- 172.21.15.136:0/2814706468 shutdown_connections 2024-04-25T11:32:58.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 -- 172.21.15.136:0/2814706468 wait complete. 2024-04-25T11:32:58.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.033+0000 7f8dbab77640 1 Processor -- start 2024-04-25T11:32:58.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- start start 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db41e3870 0x7f8db4150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 0x7f8db4148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8db40ca870 con 0x7f8db41e0600 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8db40c7340 con 0x7f8db41e0a10 2024-04-25T11:32:58.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8db40ca9f0 con 0x7f8db41e3870 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43326/0 (socket says 172.21.15.136:43326) 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 -- 172.21.15.136:0/2322569623 learned_addr learned my addr 172.21.15.136:0/2322569623 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db9114640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 0x7f8db4148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db8913640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db41e3870 0x7f8db4150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 -- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db41e3870 msgr2=0x7f8db4150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db41e3870 0x7f8db4150cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 -- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 msgr2=0x7f8db4148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 0x7f8db4148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 -- 172.21.15.136:0/2322569623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db41e00c0 con 0x7f8db41e0600 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db9114640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 0x7f8db4148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db3fff640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8da8010b40 tx=0x7f8da80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:58.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da801b070 con 0x7f8db41e0600 2024-04-25T11:32:58.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da8011040 con 0x7f8db41e0600 2024-04-25T11:32:58.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da800ee30 con 0x7f8db41e0600 2024-04-25T11:32:58.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db41e03a0 con 0x7f8db41e0600 2024-04-25T11:32:58.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db41f3470 con 0x7f8db41e0600 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db406a350 con 0x7f8db41e0600 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.037+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8da801f3f0 con 0x7f8db41e0600 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.041+0000 7f8db17fa640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 0x7f8da0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.041+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f8da8091f80 con 0x7f8db41e0600 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.041+0000 7f8db8913640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 0x7f8da0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:32:58.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.041+0000 7f8db8913640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 0x7f8da0071910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8da40af000 tx=0x7f8da4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:32:58.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.045+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8da805e330 con 0x7f8db41e0600 2024-04-25T11:32:58.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8db4071300 con 0x7f8db41e0600 2024-04-25T11:32:58.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8db17fa640 1 -- 172.21.15.136:0/2322569623 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6099 (secure 0 0 0) 0x7f8da8062630 con 0x7f8db41e0600 2024-04-25T11:32:58.221 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:32:58.221 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:32:54.704787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5036,"qdb_cluster":[5385,5238,5517,5036]},"id":1}]} 2024-04-25T11:32:58.222 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 53 2024-04-25T11:32:58.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 msgr2=0x7f8da0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 0x7f8da0071910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8da40af000 tx=0x7f8da4008040 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 msgr2=0x7f8db4147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8da8010b40 tx=0x7f8da80057f0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 shutdown_connections 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8db41e0a10 0x7f8db4148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8db41e0600 0x7f8db4147f10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8da006f400 0x7f8da0071910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 --2- 172.21.15.136:0/2322569623 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8db41e3870 0x7f8db4150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 >> 172.21.15.136:0/2322569623 conn(0x7f8db40c9d50 msgr2=0x7f8db40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 shutdown_connections 2024-04-25T11:32:58.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:32:58.237+0000 7f8dbab77640 1 -- 172.21.15.136:0/2322569623 wait complete. 2024-04-25T11:32:58.243 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:00.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:00.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.413+0000 7f977fa19640 1 Processor -- start 2024-04-25T11:33:00.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.413+0000 7f977fa19640 1 -- start start 2024-04-25T11:33:00.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977fa19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9778164a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977fa19640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97780c8090 con 0x7f9778158da0 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977fa19640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97780c6aa0 con 0x7f9778153190 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977fa19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97780c8370 con 0x7f9778162660 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9778158da0 0x7f97781591b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43676/0 (socket says 172.21.15.136:43676) 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 -- 172.21.15.136:0/2750383378 learned_addr learned my addr 172.21.15.136:0/2750383378 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977dfb6640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9778164a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949639533 0 0) 0x7f97780c8090 con 0x7f9778158da0 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9764003530 con 0x7f9778158da0 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003687602 0 0) 0x7f97780c6aa0 con 0x7f9778153190 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97780c8090 con 0x7f9778153190 2024-04-25T11:33:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1075172249 0 0) 0x7f9764003530 con 0x7f9778158da0 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 msgr2=0x7f9778164a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9778164a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 >> v1:172.21.15.157:6789/0 conn(0x7f9778153190 legacy=0x7f97780c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97780c6d80 con 0x7f9778158da0 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f9768001680 con 0x7f9778158da0 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 >> v1:172.21.15.136:6789/0 conn(0x7f9778158da0 legacy=0x7f97781591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9764003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97640040e0 0x7f9764004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9764004d50 0x7f9764005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f97780c8370 con 0x7f97640040e0 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f976400dd40 con 0x7f9764004d50 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f976400df40 con 0x7f9778162660 2024-04-25T11:33:00.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97640040e0 0x7f9764004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977dfb6640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9764004d50 0x7f9764005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977d7b5640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9764003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 -- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 msgr2=0x7f9764003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9764003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 -- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9764004d50 msgr2=0x7f9764005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9764004d50 0x7f9764005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977cfb4640 1 -- 172.21.15.136:0/2750383378 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9764003530 con 0x7f97640040e0 2024-04-25T11:33:00.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.417+0000 7f977d7b5640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778162660 0x7f9764003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:00.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97640040e0 0x7f9764004810 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f976800c1c0 tx=0x7f976801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cab3ebe4ac2e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:00.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007cf0 con 0x7f97640040e0 2024-04-25T11:33:00.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9768005ae0 con 0x7f97640040e0 2024-04-25T11:33:00.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f976ffff640 1 -- 172.21.15.136:0/2750383378 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007070 con 0x7f97640040e0 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 -- 172.21.15.136:0/2750383378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97640040e0 msgr2=0x7f9764004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 --2- 172.21.15.136:0/2750383378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97640040e0 0x7f9764004810 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f976800c1c0 tx=0x7f976801f390 comp rx=0 tx=0).stop 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977dfb6640 1 -- 172.21.15.136:0/2750383378 reap_dead start 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 -- 172.21.15.136:0/2750383378 shutdown_connections 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 -- 172.21.15.136:0/2750383378 >> 172.21.15.136:0/2750383378 conn(0x7f97780c9d50 msgr2=0x7f97780ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 -- 172.21.15.136:0/2750383378 shutdown_connections 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.421+0000 7f977fa19640 1 -- 172.21.15.136:0/2750383378 wait complete. 2024-04-25T11:33:00.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 Processor -- start 2024-04-25T11:33:00.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- start start 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97640040e0 0x7f97780cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778142e50 0x7f9778138c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f97780ca870 con 0x7f97780ccd60 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97780c7340 con 0x7f97640040e0 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97780ca9f0 con 0x7f9778142e50 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43332/0 (socket says 172.21.15.136:43332) 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 -- 172.21.15.136:0/2538100874 learned_addr learned my addr 172.21.15.136:0/2538100874 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:00.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977d7b5640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97640040e0 0x7f97780cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977dfb6640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778142e50 0x7f9778138c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 -- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778142e50 msgr2=0x7f9778138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778142e50 0x7f9778138c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 -- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97640040e0 msgr2=0x7f97780cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97640040e0 0x7f97780cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 -- 172.21.15.136:0/2538100874 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97780cb7f0 con 0x7f97780ccd60 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977cfb4640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9768026000 tx=0x7f9764005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768005c90 con 0x7f97780ccd60 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9768005e30 con 0x7f97780ccd60 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768004a10 con 0x7f97780ccd60 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97780cb230 con 0x7f97780ccd60 2024-04-25T11:33:00.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.425+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9778141980 con 0x7f97780ccd60 2024-04-25T11:33:00.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977813d0d0 con 0x7f97780ccd60 2024-04-25T11:33:00.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f97680ba030 con 0x7f97780ccd60 2024-04-25T11:33:00.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f976dffb640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 0x7f975c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:00.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f976812c380 con 0x7f97780ccd60 2024-04-25T11:33:00.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f977d7b5640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 0x7f975c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:00.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.429+0000 7f977d7b5640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 0x7f975c071940 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9774000f70 tx=0x7f9774009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:00.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.433+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f97680190a0 con 0x7f97780ccd60 2024-04-25T11:33:00.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.629+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f977806a350 con 0x7f97780ccd60 2024-04-25T11:33:00.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f976dffb640 1 -- 172.21.15.136:0/2538100874 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6104 (secure 0 0 0) 0x7f97680f8730 con 0x7f97780ccd60 2024-04-25T11:33:00.617 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:00.617 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5036,5517,5238,5356,5385]},"id":1}]} 2024-04-25T11:33:00.617 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 54 2024-04-25T11:33:00.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 msgr2=0x7f975c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 0x7f975c071940 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9774000f70 tx=0x7f9774009040 comp rx=0 tx=0).stop 2024-04-25T11:33:00.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 msgr2=0x7f977813d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:00.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9768026000 tx=0x7f9764005840 comp rx=0 tx=0).stop 2024-04-25T11:33:00.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 shutdown_connections 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f975c06f430 0x7f975c071940 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9778142e50 0x7f9778138c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97780ccd60 0x7f977813d790 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 --2- 172.21.15.136:0/2538100874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97640040e0 0x7f97780cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 >> 172.21.15.136:0/2538100874 conn(0x7f97780c9d50 msgr2=0x7f97780732b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 shutdown_connections 2024-04-25T11:33:00.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:00.633+0000 7f977fa19640 1 -- 172.21.15.136:0/2538100874 wait complete. 2024-04-25T11:33:00.641 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4303 2024-04-25T11:33:00.641 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2024-04-25T11:33:02.642 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-25T11:33:02.642 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-25T11:33:02.643 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-25T11:33:02.645 INFO:tasks.ceph.mds.f:Started 2024-04-25T11:33:02.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:02.753 INFO:tasks.ceph.mds.f.smithi157.stdout:starting mds.f at 2024-04-25T11:33:02.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 Processor -- start 2024-04-25T11:33:02.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 -- start start 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd364143170 0x7fd36415d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd364070870 0x7fd3640c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd364143580 con 0x7fd364070c80 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd36415dd20 con 0x7fd364070870 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36c316640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd36415e000 con 0x7fd364143170 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd364070870 0x7fd3640c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd364070870 0x7fd3640c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43348/0 (socket says 172.21.15.136:43348) 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 -- 172.21.15.136:0/2813375323 learned_addr learned my addr 172.21.15.136:0/2813375323 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36a8b3640 1 --1- 172.21.15.136:0/2813375323 >> v1:172.21.15.136:6789/0 conn(0x7fd364070c80 0x7fd364159e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43686/0 (socket says 172.21.15.136:43686) 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd36a0b2640 1 --2- 172.21.15.136:0/2813375323 >> v2:172.21.15.157:3300/0 conn(0x7fd364143170 0x7fd36415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 -- 172.21.15.136:0/2813375323 >> v2:172.21.15.157:3300/0 conn(0x7fd364143170 msgr2=0x7fd36415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 --2- 172.21.15.136:0/2813375323 >> v2:172.21.15.157:3300/0 conn(0x7fd364143170 0x7fd36415d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 -- 172.21.15.136:0/2813375323 >> v1:172.21.15.136:6789/0 conn(0x7fd364070c80 legacy=0x7fd364159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3690b0640 1 -- 172.21.15.136:0/2813375323 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388543052 0 0) 0x7fd35c0014a0 con 0x7fd364070c80 2024-04-25T11:33:02.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.825+0000 7fd3698b1640 1 -- 172.21.15.136:0/2813375323 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36406fd90 con 0x7fd364070870 2024-04-25T11:33:02.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd3698b1640 1 --2- 172.21.15.136:0/2813375323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364070870 0x7fd3640c6960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd3640c9cb0 tx=0x7fd35401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d759c2552e80bdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:02.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd3690b0640 1 -- 172.21.15.136:0/2813375323 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540b5070 con 0x7fd364070870 2024-04-25T11:33:02.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd3690b0640 1 -- 172.21.15.136:0/2813375323 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3540a8a10 con 0x7fd364070870 2024-04-25T11:33:02.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd3690b0640 1 -- 172.21.15.136:0/2813375323 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540a8de0 con 0x7fd364070870 2024-04-25T11:33:02.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 -- 172.21.15.136:0/2813375323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364070870 msgr2=0x7fd3640c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 --2- 172.21.15.136:0/2813375323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364070870 0x7fd3640c6960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd3640c9cb0 tx=0x7fd35401ed10 comp rx=0 tx=0).stop 2024-04-25T11:33:02.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 -- 172.21.15.136:0/2813375323 shutdown_connections 2024-04-25T11:33:02.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 --2- 172.21.15.136:0/2813375323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364070870 0x7fd3640c6960 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 --2- 172.21.15.136:0/2813375323 >> v2:172.21.15.157:3300/0 conn(0x7fd364143170 0x7fd36415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 -- 172.21.15.136:0/2813375323 >> 172.21.15.136:0/2813375323 conn(0x7fd3640c9d50 msgr2=0x7fd3640ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:02.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 -- 172.21.15.136:0/2813375323 shutdown_connections 2024-04-25T11:33:02.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.829+0000 7fd36c316640 1 -- 172.21.15.136:0/2813375323 wait complete. 2024-04-25T11:33:02.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 Processor -- start 2024-04-25T11:33:02.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- start start 2024-04-25T11:33:02.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd364070870 0x7fd36415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd364070c80 0x7fd36415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd36415e2e0 con 0x7fd364143170 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd364070350 con 0x7fd364070c80 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd3640712a0 con 0x7fd364070870 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43356/0 (socket says 172.21.15.136:43356) 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 -- 172.21.15.136:0/1889270281 learned_addr learned my addr 172.21.15.136:0/1889270281 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a0b2640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd364070870 0x7fd36415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd3698b1640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd364070c80 0x7fd36415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 -- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd364070870 msgr2=0x7fd36415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd364070870 0x7fd36415c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 -- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd364070c80 msgr2=0x7fd36415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd364070c80 0x7fd36415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:02.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 -- 172.21.15.136:0/1889270281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36415cdc0 con 0x7fd364143170 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36a8b3640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c6d0 tx=0x7fd35c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c018070 con 0x7fd364143170 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd35c004d70 con 0x7fd364143170 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c00a880 con 0x7fd364143170 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36415afe0 con 0x7fd364143170 2024-04-25T11:33:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36415bac0 con 0x7fd364143170 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.833+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36406a350 con 0x7fd364143170 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.837+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd35c00ac30 con 0x7fd364143170 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.837+0000 7fd35affd640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 0x7fd348071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.837+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7fd35c08ee00 con 0x7fd364143170 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.837+0000 7fd36a0b2640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 0x7fd348071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:02.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.837+0000 7fd36a0b2640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 0x7fd348071940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd36000b930 tx=0x7fd360008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:02.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:02.841+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd364143580 con 0x7fd364143170 2024-04-25T11:33:03.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.033+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3641f3410 con 0x7fd364143170 2024-04-25T11:33:03.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.033+0000 7fd35affd640 1 -- 172.21.15.136:0/1889270281 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6104 (secure 0 0 0) 0x7fd35c05b800 con 0x7fd364143170 2024-04-25T11:33:03.019 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:03.019 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5036,5517,5238,5356,5385]},"id":1}]} 2024-04-25T11:33:03.019 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 54 2024-04-25T11:33:03.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 msgr2=0x7fd348071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:03.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 0x7fd348071940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd36000b930 tx=0x7fd360008040 comp rx=0 tx=0).stop 2024-04-25T11:33:03.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 msgr2=0x7fd3640bf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:03.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00c6d0 tx=0x7fd35c00cb90 comp rx=0 tx=0).stop 2024-04-25T11:33:03.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 shutdown_connections 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd364143170 0x7fd3640bf240 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd364070c80 0x7fd36415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd34806f430 0x7fd348071940 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 --2- 172.21.15.136:0/1889270281 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd364070870 0x7fd36415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 >> 172.21.15.136:0/1889270281 conn(0x7fd3640c9d50 msgr2=0x7fd3640c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 shutdown_connections 2024-04-25T11:33:03.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:03.037+0000 7fd36c316640 1 -- 172.21.15.136:0/1889270281 wait complete. 2024-04-25T11:33:03.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-04-25T11:33:05.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:05.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 Processor -- start 2024-04-25T11:33:05.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 -- start start 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efd000ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0015c6f0 con 0x7efd0015c0d0 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0015c990 con 0x7efd00160370 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efd05f69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd000c6310 con 0x7efd0015ff60 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efcff7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7efd0015c0d0 0x7efd000cb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59044/0 (socket says 172.21.15.136:59044) 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efcff7fe640 1 -- 172.21.15.136:0/3387067387 learned_addr learned my addr 172.21.15.136:0/3387067387 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efcfeffd640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efd000ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197340700 0 0) 0x7efd0015c6f0 con 0x7efd0015c0d0 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.217+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efce8003530 con 0x7efd0015c0d0 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3796816256 0 0) 0x7efce8003530 con 0x7efd0015c0d0 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 msgr2=0x7efd000ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efd000ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 >> v1:172.21.15.157:6789/0 conn(0x7efd00160370 legacy=0x7efd00166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0015cc30 con 0x7efd0015c0d0 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7efcf8001a00 con 0x7efd0015c0d0 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 >> v1:172.21.15.136:6789/0 conn(0x7efd0015c0d0 legacy=0x7efd000cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efce80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce8003f70 0x7efce8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efce8004d50 0x7efce8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efd000c6310 con 0x7efce8004d50 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efce800bb60 con 0x7efce8003f70 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efce800bd60 con 0x7efd0015ff60 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efce8004d50 0x7efce8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcff7fe640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efce80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfeffd640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce8003f70 0x7efce8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 -- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 msgr2=0x7efce80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efce80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 -- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce8003f70 msgr2=0x7efce8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce8003f70 0x7efce8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 -- 172.21.15.136:0/3387067387 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce8003530 con 0x7efce8004d50 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcff7fe640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0015ff60 0x7efce80039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:05.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfeffd640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efce8003f70 0x7efce8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfffff640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efce8004d50 0x7efce8005160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efd00159c30 tx=0x7efcf001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ac30738893850f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00013a0 con 0x7efce8004d50 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf00052f0 con 0x7efce8004d50 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efcfe7fc640 1 -- 172.21.15.136:0/3387067387 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0003840 con 0x7efce8004d50 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efd05f69640 1 -- 172.21.15.136:0/3387067387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efce8004d50 msgr2=0x7efce8005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.221+0000 7efd05f69640 1 --2- 172.21.15.136:0/3387067387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efce8004d50 0x7efce8005160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efd00159c30 tx=0x7efcf001c3c0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfffff640 1 -- 172.21.15.136:0/3387067387 reap_dead start 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- 172.21.15.136:0/3387067387 shutdown_connections 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- 172.21.15.136:0/3387067387 >> 172.21.15.136:0/3387067387 conn(0x7efd0006f9d0 msgr2=0x7efd0006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- 172.21.15.136:0/3387067387 shutdown_connections 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- 172.21.15.136:0/3387067387 wait complete. 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 Processor -- start 2024-04-25T11:33:05.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- start start 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd001fa1a0 0x7efd001554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 0x7efd000cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efd0015cf10 con 0x7efd00148df0 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd0015fa40 con 0x7efd001fa1a0 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efd05f69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd000c9170 con 0x7efd00149200 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41854/0 (socket says 172.21.15.136:41854) 2024-04-25T11:33:05.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 -- 172.21.15.136:0/2540817683 learned_addr learned my addr 172.21.15.136:0/2540817683 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcff7fe640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd001fa1a0 0x7efd001554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfffff640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 0x7efd000cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 -- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 msgr2=0x7efd000cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 0x7efd000cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 -- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd001fa1a0 msgr2=0x7efd001554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd001fa1a0 0x7efd001554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 -- 172.21.15.136:0/2540817683 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd001fa5b0 con 0x7efd00148df0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfffff640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 0x7efd000cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.225+0000 7efcfeffd640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efcf4009160 tx=0x7efcf4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4004220 con 0x7efd00148df0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf40043c0 con 0x7efd00148df0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd00155c20 con 0x7efd00148df0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4015890 con 0x7efd00148df0 2024-04-25T11:33:05.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd00153cb0 con 0x7efd00148df0 2024-04-25T11:33:05.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0006a350 con 0x7efd00148df0 2024-04-25T11:33:05.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efcf4015a30 con 0x7efd00148df0 2024-04-25T11:33:05.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 0x7efcec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7efcf408fe00 con 0x7efd00148df0 2024-04-25T11:33:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efcff7fe640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 0x7efcec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:05.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.229+0000 7efcff7fe640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 0x7efcec071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efcf8007270 tx=0x7efcf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:05.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.233+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efd001fa5b0 con 0x7efd00148df0 2024-04-25T11:33:05.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.441+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd00071d80 con 0x7efd00148df0 2024-04-25T11:33:05.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efce7fff640 1 -- 172.21.15.136:0/2540817683 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7efcf4019030 con 0x7efd00148df0 2024-04-25T11:33:05.428 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:05.428 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:05.429 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 msgr2=0x7efcec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 0x7efcec071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efcf8007270 tx=0x7efcf8009040 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 msgr2=0x7efd00149890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efcf4009160 tx=0x7efcf4011a60 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 shutdown_connections 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efcec06f400 0x7efcec071910 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd00149200 0x7efd000cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd00148df0 0x7efd00149890 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 --2- 172.21.15.136:0/2540817683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd001fa1a0 0x7efd001554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 >> 172.21.15.136:0/2540817683 conn(0x7efd0006f9d0 msgr2=0x7efd000c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 shutdown_connections 2024-04-25T11:33:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:05.445+0000 7efd05f69640 1 -- 172.21.15.136:0/2540817683 wait complete. 2024-04-25T11:33:05.453 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-25T11:33:05.453 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2024-04-25T11:33:10.155 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:10.168+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T11:33:10.156 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:10.168+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:33:10.156 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:10.168+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:33:10.156 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:10.168+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:33:10.156 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:10.168+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:33:12.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:12.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.653+0000 7f03ac6c7640 1 Processor -- start 2024-04-25T11:33:12.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.653+0000 7f03ac6c7640 1 -- start start 2024-04-25T11:33:12.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03ac6c7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f03a40c96e0 0x7f03a41540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03ac6c7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f03a415aae0 0x7f03a40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03ac6c7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03a40c9af0 con 0x7f03a40c79e0 2024-04-25T11:33:12.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03ac6c7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f03a406cc10 con 0x7f03a40c96e0 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03ac6c7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f03a40cb220 con 0x7f03a415aae0 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f03a40c96e0 0x7f03a41540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9c62640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f03a40c79e0 0x7f03a415a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:59054/0 (socket says 172.21.15.136:59054) 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9c62640 1 -- 172.21.15.136:0/1864471481 learned_addr learned my addr 172.21.15.136:0/1864471481 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aac64640 1 --2- 172.21.15.136:0/1864471481 >> v2:172.21.15.157:3300/0 conn(0x7f03a415aae0 0x7f03a40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 -- 172.21.15.136:0/1864471481 >> v2:172.21.15.157:3300/0 conn(0x7f03a415aae0 msgr2=0x7f03a40c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 --2- 172.21.15.136:0/1864471481 >> v2:172.21.15.157:3300/0 conn(0x7f03a415aae0 0x7f03a40c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 -- 172.21.15.136:0/1864471481 >> v1:172.21.15.136:6789/0 conn(0x7f03a40c79e0 legacy=0x7f03a415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 -- 172.21.15.136:0/1864471481 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a40cb500 con 0x7f03a40c96e0 2024-04-25T11:33:12.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9461640 1 -- 172.21.15.136:0/1864471481 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713426103 0 0) 0x7f03a40c9af0 con 0x7f03a40c79e0 2024-04-25T11:33:12.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03aa463640 1 --2- 172.21.15.136:0/1864471481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a40c96e0 0x7f03a41540d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f03a4075140 tx=0x7f03a001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ae0615796c254d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:12.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9461640 1 -- 172.21.15.136:0/1864471481 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a00b5070 con 0x7f03a40c96e0 2024-04-25T11:33:12.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9461640 1 -- 172.21.15.136:0/1864471481 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a00a8970 con 0x7f03a40c96e0 2024-04-25T11:33:12.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.657+0000 7f03a9461640 1 -- 172.21.15.136:0/1864471481 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a00a8d40 con 0x7f03a40c96e0 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- 172.21.15.136:0/1864471481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a40c96e0 msgr2=0x7f03a41540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/1864471481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a40c96e0 0x7f03a41540d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f03a4075140 tx=0x7f03a001ec70 comp rx=0 tx=0).stop 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- 172.21.15.136:0/1864471481 shutdown_connections 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/1864471481 >> v2:172.21.15.157:3300/0 conn(0x7f03a415aae0 0x7f03a40c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/1864471481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a40c96e0 0x7f03a41540d0 secure :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f03a4075140 tx=0x7f03a001ec70 comp rx=0 tx=0).stop 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- 172.21.15.136:0/1864471481 >> 172.21.15.136:0/1864471481 conn(0x7f03a406f9d0 msgr2=0x7f03a406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- 172.21.15.136:0/1864471481 shutdown_connections 2024-04-25T11:33:12.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- 172.21.15.136:0/1864471481 wait complete. 2024-04-25T11:33:12.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 Processor -- start 2024-04-25T11:33:12.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 -- start start 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.661+0000 7f03ac6c7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03a40c79e0 0x7f03a4148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03a40c57d0 0x7f03a41e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40cf0f0 con 0x7f03a415aae0 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4154610 con 0x7f03a40c57d0 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03a40ced40 con 0x7f03a40c79e0 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41862/0 (socket says 172.21.15.136:41862) 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 -- 172.21.15.136:0/370248370 learned_addr learned my addr 172.21.15.136:0/370248370 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:12.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03aa463640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03a40c79e0 0x7f03a4148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03aac64640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03a40c57d0 0x7f03a41e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 -- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03a40c79e0 msgr2=0x7f03a4148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03a40c79e0 0x7f03a4148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 -- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03a40c57d0 msgr2=0x7f03a41e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03a40c57d0 0x7f03a41e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 -- 172.21.15.136:0/370248370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a41f5bb0 con 0x7f03a415aae0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03a9c62640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f039400c480 tx=0x7f039400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394018070 con 0x7f03a415aae0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0394007500 con 0x7f03a415aae0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a41f5e90 con 0x7f03a415aae0 2024-04-25T11:33:12.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394004a30 con 0x7f03a415aae0 2024-04-25T11:33:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41f6450 con 0x7f03a415aae0 2024-04-25T11:33:12.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a406a350 con 0x7f03a415aae0 2024-04-25T11:33:12.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f039401b070 con 0x7f03a415aae0 2024-04-25T11:33:12.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 0x7f0388071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:12.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f039408eb10 con 0x7f03a415aae0 2024-04-25T11:33:12.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.665+0000 7f03aa463640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 0x7f0388071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:12.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.669+0000 7f03aa463640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 0x7f0388071940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03a00b2000 tx=0x7f03a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:12.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.673+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f039405b510 con 0x7f03a415aae0 2024-04-25T11:33:12.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03a41579a0 con 0x7f03a415aae0 2024-04-25T11:33:12.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f039affd640 1 -- 172.21.15.136:0/370248370 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7f039405ae10 con 0x7f03a415aae0 2024-04-25T11:33:12.916 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:12.917 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:12.917 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:12.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 msgr2=0x7f0388071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 0x7f0388071940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03a00b2000 tx=0x7f03a0008040 comp rx=0 tx=0).stop 2024-04-25T11:33:12.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 msgr2=0x7f03a41f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:12.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f039400c480 tx=0x7f039400c940 comp rx=0 tx=0).stop 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 shutdown_connections 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03a40c57d0 0x7f03a41e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03a415aae0 0x7f03a41f26a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038806f430 0x7f0388071940 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.933+0000 7f03ac6c7640 1 --2- 172.21.15.136:0/370248370 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03a40c79e0 0x7f03a4148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.937+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 >> 172.21.15.136:0/370248370 conn(0x7f03a406f9d0 msgr2=0x7f03a4152540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.937+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 shutdown_connections 2024-04-25T11:33:12.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:12.937+0000 7f03ac6c7640 1 -- 172.21.15.136:0/370248370 wait complete. 2024-04-25T11:33:12.946 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2024-04-25T11:33:12.947 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T11:33:12.947 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:33:12.964 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T11:33:12.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 Processor -- start 2024-04-25T11:33:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 -- start start 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff65c160370 0x7ff65c166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65c15c6f0 con 0x7ff65c15ff60 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff65c0c6310 con 0x7ff65c160370 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff663eaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c15c990 con 0x7ff65c15c0d0 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff661c4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff66244c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff65c160370 0x7ff65c166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff661c4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:49558/0 (socket says 172.21.15.136:49558) 2024-04-25T11:33:13.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.149+0000 7ff661c4b640 1 -- 172.21.15.136:0/604392696 learned_addr learned my addr 172.21.15.136:0/604392696 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff66244c640 1 -- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 msgr2=0x7ff65c0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff66244c640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff66244c640 1 -- 172.21.15.136:0/604392696 >> v1:172.21.15.157:6789/0 conn(0x7ff65c15ff60 legacy=0x7ff65c0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff66244c640 1 -- 172.21.15.136:0/604392696 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c15cc30 con 0x7ff65c160370 2024-04-25T11:33:13.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff661c4b640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:33:13.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff66244c640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c166a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff654009d90 tx=0x7ff65401f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38c9918b8fbf325f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff660c49640 1 -- 172.21.15.136:0/604392696 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540b4070 con 0x7ff65c160370 2024-04-25T11:33:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff660c49640 1 -- 172.21.15.136:0/604392696 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6540af600 con 0x7ff65c160370 2024-04-25T11:33:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff660c49640 1 -- 172.21.15.136:0/604392696 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540af9d0 con 0x7ff65c160370 2024-04-25T11:33:13.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff663eaf640 1 -- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 msgr2=0x7ff65c166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.153+0000 7ff663eaf640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c166a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff654009d90 tx=0x7ff65401f150 comp rx=0 tx=0).stop 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- 172.21.15.136:0/604392696 shutdown_connections 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c166a50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 --2- 172.21.15.136:0/604392696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15c0d0 0x7ff65c0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- 172.21.15.136:0/604392696 >> 172.21.15.136:0/604392696 conn(0x7ff65c06f9d0 msgr2=0x7ff65c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- 172.21.15.136:0/604392696 shutdown_connections 2024-04-25T11:33:13.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- 172.21.15.136:0/604392696 wait complete. 2024-04-25T11:33:13.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 Processor -- start 2024-04-25T11:33:13.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- start start 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 0x7ff65c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c0cbc60 con 0x7ff65c15c0d0 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c15fa40 con 0x7ff65c160370 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff663eaf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c15cf10 con 0x7ff65c15ff60 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41864/0 (socket says 172.21.15.136:41864) 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 -- 172.21.15.136:0/2753490932 learned_addr learned my addr 172.21.15.136:0/2753490932 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff66144a640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 0x7ff65c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff66244c640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 -- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 msgr2=0x7ff65c1f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 0x7ff65c1f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 -- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 msgr2=0x7ff65c147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff661c4b640 1 -- 172.21.15.136:0/2753490932 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c150a70 con 0x7ff65c15c0d0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.157+0000 7ff66144a640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 0x7ff65c1f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff661c4b640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff65800e960 tx=0x7ff65800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658017070 con 0x7ff65c15c0d0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65800c9a0 con 0x7ff65c15c0d0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65800cd50 con 0x7ff65c15c0d0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c1f6770 con 0x7ff65c15c0d0 2024-04-25T11:33:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c155330 con 0x7ff65c15c0d0 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c06a350 con 0x7ff65c15c0d0 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff65801a070 con 0x7ff65c15c0d0 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 0x7ff640071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:13.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7ff65808fb10 con 0x7ff65c15c0d0 2024-04-25T11:33:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.161+0000 7ff66144a640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 0x7ff640071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:13.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.165+0000 7ff66144a640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 0x7ff640071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff65c15ab50 tx=0x7ff64c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:13.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.165+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff65805bd20 con 0x7ff65c15c0d0 2024-04-25T11:33:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff65c159f10 con 0x7ff65c15c0d0 2024-04-25T11:33:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff6527fc640 1 -- 172.21.15.136:0/2753490932 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7ff658060020 con 0x7ff65c15c0d0 2024-04-25T11:33:13.396 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:13.396 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:13.397 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:13.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 msgr2=0x7ff640071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.398 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 0x7ff640071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff65c15ab50 tx=0x7ff64c009040 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 msgr2=0x7ff65c150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff65800e960 tx=0x7ff65800ee20 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 shutdown_connections 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff65c160370 0x7ff65c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff64006f400 0x7ff640071910 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff65c15ff60 0x7ff65c1f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 --2- 172.21.15.136:0/2753490932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff65c15c0d0 0x7ff65c150530 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 >> 172.21.15.136:0/2753490932 conn(0x7ff65c06f9d0 msgr2=0x7ff65c15b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 shutdown_connections 2024-04-25T11:33:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:13.413+0000 7ff663eaf640 1 -- 172.21.15.136:0/2753490932 wait complete. 2024-04-25T11:33:13.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:15.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 Processor -- start 2024-04-25T11:33:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 -- start start 2024-04-25T11:33:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff9d815aae0 0x7ff9d80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d80c9af0 con 0x7ff9d80c79e0 2024-04-25T11:33:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d80cb220 con 0x7ff9d80c96e0 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9df9b1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d806cc10 con 0x7ff9d815aae0 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff9d815aae0 0x7ff9d80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff9d815aae0 0x7ff9d80c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47590/0 (socket says 172.21.15.136:47590) 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 -- 172.21.15.136:0/4096117591 learned_addr learned my addr 172.21.15.136:0/4096117591 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 -- 172.21.15.136:0/4096117591 >> v1:172.21.15.157:6789/0 conn(0x7ff9d80c96e0 legacy=0x7ff9d81540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 -- 172.21.15.136:0/4096117591 >> v1:172.21.15.136:6789/0 conn(0x7ff9d80c79e0 legacy=0x7ff9d815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.597+0000 7ff9ddf4e640 1 -- 172.21.15.136:0/4096117591 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d80cb500 con 0x7ff9d815aae0 2024-04-25T11:33:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9ddf4e640 1 --2- 172.21.15.136:0/4096117591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d815aae0 0x7ff9d80c7df0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff9d806cd90 tx=0x7ff9d001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123a229a84836482 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9cffff640 1 -- 172.21.15.136:0/4096117591 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00a5690 con 0x7ff9d815aae0 2024-04-25T11:33:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9cffff640 1 -- 172.21.15.136:0/4096117591 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d00a5830 con 0x7ff9d815aae0 2024-04-25T11:33:15.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9cffff640 1 -- 172.21.15.136:0/4096117591 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00a5c00 con 0x7ff9d815aae0 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 -- 172.21.15.136:0/4096117591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d815aae0 msgr2=0x7ff9d80c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 --2- 172.21.15.136:0/4096117591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d815aae0 0x7ff9d80c7df0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff9d806cd90 tx=0x7ff9d001c9d0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 -- 172.21.15.136:0/4096117591 shutdown_connections 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 --2- 172.21.15.136:0/4096117591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d815aae0 0x7ff9d80c7df0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 -- 172.21.15.136:0/4096117591 >> 172.21.15.136:0/4096117591 conn(0x7ff9d806f9d0 msgr2=0x7ff9d806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 -- 172.21.15.136:0/4096117591 shutdown_connections 2024-04-25T11:33:15.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.601+0000 7ff9df9b1640 1 -- 172.21.15.136:0/4096117591 wait complete. 2024-04-25T11:33:15.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.605+0000 7ff9df9b1640 1 Processor -- start 2024-04-25T11:33:15.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.605+0000 7ff9df9b1640 1 -- start start 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9d80c79e0 0x7ff9d81f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9d815aae0 0x7ff9d8149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d806d5c0 con 0x7ff9d80c96e0 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d806d440 con 0x7ff9d80c79e0 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d8168540 con 0x7ff9d815aae0 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47602/0 (socket says 172.21.15.136:47602) 2024-04-25T11:33:15.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 -- 172.21.15.136:0/2209834699 learned_addr learned my addr 172.21.15.136:0/2209834699 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dd74d640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9d80c79e0 0x7ff9d81f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9ddf4e640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9d815aae0 0x7ff9d8149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 -- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9d815aae0 msgr2=0x7ff9d8149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9d815aae0 0x7ff9d8149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 -- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9d80c79e0 msgr2=0x7ff9d81f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9d80c79e0 0x7ff9d81f4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 -- 172.21.15.136:0/2209834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d8144680 con 0x7ff9d80c96e0 2024-04-25T11:33:15.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9dcf4c640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c370 tx=0x7ff9c800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:15.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80052d0 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c8005470 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 -- 172.21.15.136:0/2209834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d8144960 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8007260 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9df9b1640 1 -- 172.21.15.136:0/2209834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d81f3ce0 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d806a350 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.609+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff9c8007400 con 0x7ff9d80c96e0 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.613+0000 7ff9cdffb640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 0x7ff9bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.613+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7ff9c808edb0 con 0x7ff9d80c96e0 2024-04-25T11:33:15.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.613+0000 7ff9dd74d640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 0x7ff9bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:15.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.613+0000 7ff9dd74d640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 0x7ff9bc071910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b810 tx=0x7ff9d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:15.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.617+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff9c805b690 con 0x7ff9d80c96e0 2024-04-25T11:33:15.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9d81579a0 con 0x7ff9d80c96e0 2024-04-25T11:33:15.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9cdffb640 1 -- 172.21.15.136:0/2209834699 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7ff9c805af90 con 0x7ff9d80c96e0 2024-04-25T11:33:15.817 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:15.817 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:15.817 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:15.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 msgr2=0x7ff9bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 0x7ff9bc071910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400b810 tx=0x7ff9d4008040 comp rx=0 tx=0).stop 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 msgr2=0x7ff9d81415f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800c370 tx=0x7ff9c800c830 comp rx=0 tx=0).stop 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 shutdown_connections 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff9bc06f400 0x7ff9bc071910 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9d815aae0 0x7ff9d8149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9d80c96e0 0x7ff9d81415f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 --2- 172.21.15.136:0/2209834699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9d80c79e0 0x7ff9d81f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 >> 172.21.15.136:0/2209834699 conn(0x7ff9d806f9d0 msgr2=0x7ff9d80ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 shutdown_connections 2024-04-25T11:33:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:15.833+0000 7ff9c37fe640 1 -- 172.21.15.136:0/2209834699 wait complete. 2024-04-25T11:33:15.840 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:17.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:17.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:17.997+0000 7f757c9d2640 1 Processor -- start 2024-04-25T11:33:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:17.997+0000 7f757c9d2640 1 -- start start 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757c9d2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75780b5810 0x7f75780b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757c9d2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f757815c070 0x7f757806d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757c9d2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75780b4480 con 0x7f757815c480 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757c9d2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f75780b4a90 con 0x7f75780b5810 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757c9d2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f757815c990 con 0x7f757815c070 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75780b5810 0x7f75780b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75780b5810 0x7f75780b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47616/0 (socket says 172.21.15.136:47616) 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 -- 172.21.15.136:0/1737911342 learned_addr learned my addr 172.21.15.136:0/1737911342 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f756dd9b640 1 --2- 172.21.15.136:0/1737911342 >> v2:172.21.15.157:3300/0 conn(0x7f757815c070 0x7f757806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 -- 172.21.15.136:0/1737911342 >> v2:172.21.15.157:3300/0 conn(0x7f757815c070 msgr2=0x7f757806d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:17.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 --2- 172.21.15.136:0/1737911342 >> v2:172.21.15.157:3300/0 conn(0x7f757815c070 0x7f757806d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 -- 172.21.15.136:0/1737911342 >> v1:172.21.15.157:6789/0 conn(0x7f757815c480 legacy=0x7f757815f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:17.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 -- 172.21.15.136:0/1737911342 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757815cc70 con 0x7f75780b5810 2024-04-25T11:33:17.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f757659c640 1 --2- 172.21.15.136:0/1737911342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75780b5810 0x7f75780b4f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f757808f0e0 tx=0x7f756801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39872ff0cea000d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.001+0000 7f7575d9b640 1 -- 172.21.15.136:0/1737911342 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680b5070 con 0x7f75780b5810 2024-04-25T11:33:17.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f7575d9b640 1 -- 172.21.15.136:0/1737911342 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75680a89d0 con 0x7f75780b5810 2024-04-25T11:33:17.987 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f7575d9b640 1 -- 172.21.15.136:0/1737911342 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75680a8da0 con 0x7f75780b5810 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- 172.21.15.136:0/1737911342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75780b5810 msgr2=0x7f75780b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 --2- 172.21.15.136:0/1737911342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75780b5810 0x7f75780b4f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f757808f0e0 tx=0x7f756801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- 172.21.15.136:0/1737911342 shutdown_connections 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 --2- 172.21.15.136:0/1737911342 >> v2:172.21.15.157:3300/0 conn(0x7f757815c070 0x7f757806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 --2- 172.21.15.136:0/1737911342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75780b5810 0x7f75780b4f20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- 172.21.15.136:0/1737911342 >> 172.21.15.136:0/1737911342 conn(0x7f75780743a0 msgr2=0x7f75780d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:17.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- 172.21.15.136:0/1737911342 shutdown_connections 2024-04-25T11:33:17.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- 172.21.15.136:0/1737911342 wait complete. 2024-04-25T11:33:17.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 Processor -- start 2024-04-25T11:33:17.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.005+0000 7f757c9d2640 1 -- start start 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75780b5810 0x7f7578153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 0x7f75780ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f757806e030 con 0x7f757815c070 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f757815cf50 con 0x7f757815c480 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75780b5460 con 0x7f75780b5810 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47630/0 (socket says 172.21.15.136:47630) 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 -- 172.21.15.136:0/2591269823 learned_addr learned my addr 172.21.15.136:0/2591269823 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757659c640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75780b5810 0x7f7578153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f7576d9d640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 0x7f75780ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 -- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75780b5810 msgr2=0x7f7578153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75780b5810 0x7f7578153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 -- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 msgr2=0x7f75780ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 0x7f75780ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 -- 172.21.15.136:0/2591269823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75780c4a90 con 0x7f757815c070 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f7576d9d640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 0x7f75780ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756dd9b640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f756000c820 tx=0x7f756000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560015070 con 0x7f757815c070 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75600079d0 con 0x7f757815c070 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757815daf0 con 0x7f757815c070 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560007dc0 con 0x7f757815c070 2024-04-25T11:33:17.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f757815f910 con 0x7f757815c070 2024-04-25T11:33:17.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f757806a350 con 0x7f757815c070 2024-04-25T11:33:17.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.009+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7560018070 con 0x7f757815c070 2024-04-25T11:33:17.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.013+0000 7f756f7fe640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 0x7f7558071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:17.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.013+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f756008f6a0 con 0x7f757815c070 2024-04-25T11:33:17.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.013+0000 7f757659c640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 0x7f7558071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:17.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.013+0000 7f757659c640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 0x7f7558071910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f75680b2000 tx=0x7f7568008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:18.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.017+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f756005c010 con 0x7f757815c070 2024-04-25T11:33:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.273+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75780cd5f0 con 0x7f757815c070 2024-04-25T11:33:18.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.273+0000 7f756f7fe640 1 -- 172.21.15.136:0/2591269823 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7f756005b910 con 0x7f757815c070 2024-04-25T11:33:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:18.259 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:18.259 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:18.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 msgr2=0x7f7558071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 0x7f7558071910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f75680b2000 tx=0x7f7568008040 comp rx=0 tx=0).stop 2024-04-25T11:33:18.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 msgr2=0x7f757815d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:18.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f756000c820 tx=0x7f756000cce0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 shutdown_connections 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f757815c480 0x7f75780ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f757815c070 0x7f757815d5b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f755806f400 0x7f7558071910 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 --2- 172.21.15.136:0/2591269823 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75780b5810 0x7f7578153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 >> 172.21.15.136:0/2591269823 conn(0x7f75780743a0 msgr2=0x7f75780cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 shutdown_connections 2024-04-25T11:33:18.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:18.277+0000 7f757c9d2640 1 -- 172.21.15.136:0/2591269823 wait complete. 2024-04-25T11:33:18.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:20.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:20.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 Processor -- start 2024-04-25T11:33:20.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 -- start start 2024-04-25T11:33:20.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 0x7f87940c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f879415aa00 0x7f8794164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87940c8d40 con 0x7f879415a5f0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f87940c8fd0 con 0x7f879415aa00 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f879b264640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87940c3880 con 0x7f8794154060 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f8793fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f879415a5f0 0x7f87940c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60718/0 (socket says 172.21.15.136:60718) 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.473+0000 7f8793fff640 1 -- 172.21.15.136:0/1109494219 learned_addr learned my addr 172.21.15.136:0/1109494219 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 0x7f87940c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799801640 1 --2- 172.21.15.136:0/1109494219 >> v2:172.21.15.157:3300/0 conn(0x7f879415aa00 0x7f8794164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 666996610 0 0) 0x7f87940c8d40 con 0x7f879415a5f0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8784003530 con 0x7f879415a5f0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3422182956 0 0) 0x7f8784003530 con 0x7f879415a5f0 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 msgr2=0x7f87940c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 0x7f87940c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 >> v2:172.21.15.157:3300/0 conn(0x7f879415aa00 msgr2=0x7f8794164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 --2- 172.21.15.136:0/1109494219 >> v2:172.21.15.157:3300/0 conn(0x7f879415aa00 0x7f8794164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879406d9b0 con 0x7f879415a5f0 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8780001680 con 0x7f879415a5f0 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 >> v1:172.21.15.136:6789/0 conn(0x7f879415a5f0 legacy=0x7f87940c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87840039c0 0x7f8784003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8784004810 0x7f8784004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 0x7f8784005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f87940c8fd0 con 0x7f87840039c0 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87940c3880 con 0x7f8784004810 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f878400ba80 con 0x7f8794154060 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87840039c0 0x7f8784003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8793fff640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8784004810 0x7f8784004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 -- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 msgr2=0x7f8784005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8794154060 0x7f8784005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 -- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8784004810 msgr2=0x7f8784004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:20.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8784004810 0x7f8784004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 -- 172.21.15.136:0/1109494219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8784003530 con 0x7f87840039c0 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f8799000640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87840039c0 0x7f8784003e50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8788009ae0 tx=0x7f878801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229c1ce963ec3d26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880b2070 con 0x7f87840039c0 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87880a56a0 con 0x7f87840039c0 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.477+0000 7f87937fe640 1 -- 172.21.15.136:0/1109494219 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880096b0 con 0x7f87840039c0 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- 172.21.15.136:0/1109494219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87840039c0 msgr2=0x7f8784003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 --2- 172.21.15.136:0/1109494219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87840039c0 0x7f8784003e50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8788009ae0 tx=0x7f878801cab0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f8799801640 1 -- 172.21.15.136:0/1109494219 reap_dead start 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- 172.21.15.136:0/1109494219 shutdown_connections 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- 172.21.15.136:0/1109494219 >> 172.21.15.136:0/1109494219 conn(0x7f87940ca5a0 msgr2=0x7f87940ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- 172.21.15.136:0/1109494219 shutdown_connections 2024-04-25T11:33:20.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- 172.21.15.136:0/1109494219 wait complete. 2024-04-25T11:33:20.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 Processor -- start 2024-04-25T11:33:20.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- start start 2024-04-25T11:33:20.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87840039c0 0x7f879414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8794148380 0x7f8794156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f879406df30 con 0x7f879414b700 2024-04-25T11:33:20.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8794154470 con 0x7f8794148380 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.481+0000 7f879b264640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f879415a0d0 con 0x7f87840039c0 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47650/0 (socket says 172.21.15.136:47650) 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 -- 172.21.15.136:0/1685342695 learned_addr learned my addr 172.21.15.136:0/1685342695 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8793fff640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8794148380 0x7f8794156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799000640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87840039c0 0x7f879414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 -- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87840039c0 msgr2=0x7f879414e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87840039c0 0x7f879414e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 -- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8794148380 msgr2=0x7f8794156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8794148380 0x7f8794156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 -- 172.21.15.136:0/1685342695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87941f1480 con 0x7f879414b700 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f8799801640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f879415ae30 tx=0x7f878c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c00bc90 con 0x7f879414b700 2024-04-25T11:33:20.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878c01a070 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879414eac0 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c014640 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87941f1110 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8794150ba0 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f878c013070 con 0x7f879414b700 2024-04-25T11:33:20.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 0x7f8778071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:20.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.485+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f878c08fa00 con 0x7f879414b700 2024-04-25T11:33:20.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.489+0000 7f8799000640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 0x7f8778071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:20.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.489+0000 7f8799000640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 0x7f8778071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87880af000 tx=0x7f87880046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:20.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.493+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8794150ba0 con 0x7f879414b700 2024-04-25T11:33:20.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.717+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f879406a350 con 0x7f879414b700 2024-04-25T11:33:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.717+0000 7f87917fa640 1 -- 172.21.15.136:0/1685342695 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7f878c062030 con 0x7f879414b700 2024-04-25T11:33:20.701 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:20.701 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 msgr2=0x7f8778071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 0x7f8778071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f87880af000 tx=0x7f87880046d0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 msgr2=0x7f8794074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f879415ae30 tx=0x7f878c00b480 comp rx=0 tx=0).stop 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 shutdown_connections 2024-04-25T11:33:20.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f879414b700 0x7f8794074150 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8794148380 0x7f8794156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f877806f400 0x7f8778071910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 --2- 172.21.15.136:0/1685342695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87840039c0 0x7f879414e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 >> 172.21.15.136:0/1685342695 conn(0x7f87940ca5a0 msgr2=0x7f8794150ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 shutdown_connections 2024-04-25T11:33:20.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:20.721+0000 7f879b264640 1 -- 172.21.15.136:0/1685342695 wait complete. 2024-04-25T11:33:20.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:22.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.921+0000 7ff444b87640 1 Processor -- start 2024-04-25T11:33:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.921+0000 7ff444b87640 1 -- start start 2024-04-25T11:33:22.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff444b87640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff440074980 0x7ff4400c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff444b87640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff4400c4170 0x7ff4401636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff444b87640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4400c3750 con 0x7ff44014f840 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff444b87640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff44006dc00 con 0x7ff440074980 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff444b87640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff44014fc50 con 0x7ff4400c4170 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff440074980 0x7ff4400c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff440074980 0x7ff4400c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47656/0 (socket says 172.21.15.136:47656) 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 -- 172.21.15.136:0/3631046577 learned_addr learned my addr 172.21.15.136:0/3631046577 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 -- 172.21.15.136:0/3631046577 >> v2:172.21.15.157:3300/0 conn(0x7ff4400c4170 msgr2=0x7ff4401636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 --2- 172.21.15.136:0/3631046577 >> v2:172.21.15.157:3300/0 conn(0x7ff4400c4170 0x7ff4401636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 -- 172.21.15.136:0/3631046577 >> v1:172.21.15.157:6789/0 conn(0x7ff44014f840 legacy=0x7ff440073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.925+0000 7ff43dd9b640 1 -- 172.21.15.136:0/3631046577 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4400c3190 con 0x7ff440074980 2024-04-25T11:33:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff43dd9b640 1 --2- 172.21.15.136:0/3631046577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff440074980 0x7ff4400c3c30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff434009870 tx=0x7ff43401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc716bb742eb7c27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:22.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff43d59a640 1 -- 172.21.15.136:0/3631046577 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4340b1070 con 0x7ff440074980 2024-04-25T11:33:22.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff43d59a640 1 -- 172.21.15.136:0/3631046577 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4340ac630 con 0x7ff440074980 2024-04-25T11:33:22.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff43d59a640 1 -- 172.21.15.136:0/3631046577 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4340a8050 con 0x7ff440074980 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- 172.21.15.136:0/3631046577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff440074980 msgr2=0x7ff4400c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 --2- 172.21.15.136:0/3631046577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff440074980 0x7ff4400c3c30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff434009870 tx=0x7ff43401c4e0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- 172.21.15.136:0/3631046577 shutdown_connections 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 --2- 172.21.15.136:0/3631046577 >> v2:172.21.15.157:3300/0 conn(0x7ff4400c4170 0x7ff4401636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 --2- 172.21.15.136:0/3631046577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff440074980 0x7ff4400c3c30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- 172.21.15.136:0/3631046577 >> 172.21.15.136:0/3631046577 conn(0x7ff4400ca5a0 msgr2=0x7ff4400ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- 172.21.15.136:0/3631046577 shutdown_connections 2024-04-25T11:33:22.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- 172.21.15.136:0/3631046577 wait complete. 2024-04-25T11:33:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 Processor -- start 2024-04-25T11:33:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.929+0000 7ff444b87640 1 -- start start 2024-04-25T11:33:22.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff440074980 0x7ff4400c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 0x7ff4401539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff440074d90 con 0x7ff44014f840 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff440159c10 con 0x7ff4400c4170 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff440153380 con 0x7ff440074980 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47666/0 (socket says 172.21.15.136:47666) 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 -- 172.21.15.136:0/1351996373 learned_addr learned my addr 172.21.15.136:0/1351996373 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43dd9b640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 0x7ff4401539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43e59c640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff440074980 0x7ff4400c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 -- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff440074980 msgr2=0x7ff4400c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff440074980 0x7ff4400c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 -- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 msgr2=0x7ff4401539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 0x7ff4401539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 -- 172.21.15.136:0/1351996373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff440154b50 con 0x7ff44014f840 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43dd9b640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 0x7ff4401539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff43ed9d640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b9e0 tx=0x7ff43800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438019070 con 0x7ff44014f840 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43800cb20 con 0x7ff44014f840 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4400c5080 con 0x7ff44014f840 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438014440 con 0x7ff44014f840 2024-04-25T11:33:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.933+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff440155e90 con 0x7ff44014f840 2024-04-25T11:33:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff44014d660 con 0x7ff44014f840 2024-04-25T11:33:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff43801c070 con 0x7ff44014f840 2024-04-25T11:33:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff42effd640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7ff43808f590 con 0x7ff44014f840 2024-04-25T11:33:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff43e59c640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.937+0000 7ff43e59c640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff43000b810 tx=0x7ff430008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:22.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:22.941+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff44014d660 con 0x7ff44014f840 2024-04-25T11:33:22.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:33:22.966 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:33:23.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.189+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff44006a350 con 0x7ff44014f840 2024-04-25T11:33:23.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.189+0000 7ff42effd640 1 -- 172.21.15.136:0/1351996373 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7ff43805bf10 con 0x7ff44014f840 2024-04-25T11:33:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:23.174 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:23.174 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:23.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 msgr2=0x7ff420071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff43000b810 tx=0x7ff430008040 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 msgr2=0x7ff440146600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b9e0 tx=0x7ff43800bea0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 shutdown_connections 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff44014f840 0x7ff440146600 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4400c4170 0x7ff4401539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff42006f400 0x7ff420071910 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 --2- 172.21.15.136:0/1351996373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff440074980 0x7ff4400c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 >> 172.21.15.136:0/1351996373 conn(0x7ff4400ca5a0 msgr2=0x7ff4400ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 shutdown_connections 2024-04-25T11:33:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:23.193+0000 7ff444b87640 1 -- 172.21.15.136:0/1351996373 wait complete. 2024-04-25T11:33:23.196 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:25.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.385+0000 7faf42359640 1 Processor -- start 2024-04-25T11:33:25.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.385+0000 7faf42359640 1 -- start start 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf42359640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faf3c153190 0x7faf3c0c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf42359640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf3c0c8090 con 0x7faf3c1594b0 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf42359640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf3c0c6aa0 con 0x7faf3c0c9570 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf42359640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faf3c0c8370 con 0x7faf3c153190 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf408f6640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faf3c1594b0 0x7faf3c163660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38542/0 (socket says 172.21.15.136:38542) 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf408f6640 1 -- 172.21.15.136:0/2100574197 learned_addr learned my addr 172.21.15.136:0/2100574197 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3bfff640 1 --2- 172.21.15.136:0/2100574197 >> v2:172.21.15.157:3300/0 conn(0x7faf3c153190 0x7faf3c0c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506181408 0 0) 0x7faf3c0c8090 con 0x7faf3c1594b0 2024-04-25T11:33:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf24003530 con 0x7faf3c1594b0 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2984050114 0 0) 0x7faf24003530 con 0x7faf3c1594b0 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 >> v2:172.21.15.157:3300/0 conn(0x7faf3c153190 msgr2=0x7faf3c0c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 --2- 172.21.15.136:0/2100574197 >> v2:172.21.15.157:3300/0 conn(0x7faf3c153190 0x7faf3c0c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 >> v1:172.21.15.157:6789/0 conn(0x7faf3c0c9570 legacy=0x7faf3c158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c0c6d80 con 0x7faf3c1594b0 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7faf34001a00 con 0x7faf3c1594b0 2024-04-25T11:33:25.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 >> v1:172.21.15.136:6789/0 conn(0x7faf3c1594b0 legacy=0x7faf3c163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf240039c0 0x7faf24003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 0x7faf24004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf240051d0 0x7faf2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0c8370 con 0x7faf24004810 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf2400c160 con 0x7faf240051d0 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf2400c360 con 0x7faf240039c0 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 0x7faf24004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf408f6640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf240051d0 0x7faf2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3bfff640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf240039c0 0x7faf24003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 -- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf240039c0 msgr2=0x7faf24003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf240039c0 0x7faf24003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 -- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf240051d0 msgr2=0x7faf2400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf240051d0 0x7faf2400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf408f6640 1 -- 172.21.15.136:0/2100574197 reap_dead start 2024-04-25T11:33:25.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.389+0000 7faf3b7fe640 1 -- 172.21.15.136:0/2100574197 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf24003530 con 0x7faf24004810 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf3b7fe640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 0x7faf24004c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faf30001780 tx=0x7faf3001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb18a78a36aa7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300013a0 con 0x7faf24004810 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf300052f0 con 0x7faf24004810 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf3affd640 1 -- 172.21.15.136:0/2100574197 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30003840 con 0x7faf24004810 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 -- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 msgr2=0x7faf24004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 0x7faf24004c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faf30001780 tx=0x7faf3001c400 comp rx=0 tx=0).stop 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 -- 172.21.15.136:0/2100574197 shutdown_connections 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 --2- 172.21.15.136:0/2100574197 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf24004810 0x7faf24004c20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 -- 172.21.15.136:0/2100574197 >> 172.21.15.136:0/2100574197 conn(0x7faf3c0c9d50 msgr2=0x7faf3c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:25.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 -- 172.21.15.136:0/2100574197 shutdown_connections 2024-04-25T11:33:25.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.393+0000 7faf42359640 1 -- 172.21.15.136:0/2100574197 wait complete. 2024-04-25T11:33:25.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 Processor -- start 2024-04-25T11:33:25.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- start start 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf24004810 0x7faf3c1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf3c146310 0x7faf3c147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0ca870 con 0x7faf3c140ac0 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0c7340 con 0x7faf3c146310 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c0ca9f0 con 0x7faf24004810 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52420/0 (socket says 172.21.15.136:52420) 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 -- 172.21.15.136:0/875201723 learned_addr learned my addr 172.21.15.136:0/875201723 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:25.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf3bfff640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf24004810 0x7faf3c1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf3b7fe640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf3c146310 0x7faf3c147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 -- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf24004810 msgr2=0x7faf3c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf24004810 0x7faf3c1423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 -- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf3c146310 msgr2=0x7faf3c147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf3c146310 0x7faf3c147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 -- 172.21.15.136:0/875201723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c143cb0 con 0x7faf3c140ac0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf408f6640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0c8df0 tx=0x7faf3400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3400ebb0 con 0x7faf3c140ac0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf34018070 con 0x7faf3c140ac0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34013420 con 0x7faf3c140ac0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3c145d50 con 0x7faf3c140ac0 2024-04-25T11:33:25.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.397+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3c141be0 con 0x7faf3c140ac0 2024-04-25T11:33:25.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf3c06a350 con 0x7faf3c140ac0 2024-04-25T11:33:25.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faf340135c0 con 0x7faf3c140ac0 2024-04-25T11:33:25.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf38ff9640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 0x7faf28071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:25.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7faf3408fbc0 con 0x7faf3c140ac0 2024-04-25T11:33:25.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf3bfff640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 0x7faf28071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:25.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.401+0000 7faf3bfff640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 0x7faf28071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf3c051a60 tx=0x7faf2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:25.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.405+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faf3405c530 con 0x7faf3c140ac0 2024-04-25T11:33:25.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.613+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf3c073eb0 con 0x7faf3c140ac0 2024-04-25T11:33:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.613+0000 7faf38ff9640 1 -- 172.21.15.136:0/875201723 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7faf3405be30 con 0x7faf3c140ac0 2024-04-25T11:33:25.599 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:25.599 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:25.599 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 msgr2=0x7faf28071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 0x7faf28071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf3c051a60 tx=0x7faf2c008040 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 msgr2=0x7faf3c140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0c8df0 tx=0x7faf3400e530 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 shutdown_connections 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf3c140ac0 0x7faf3c140ed0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf3c146310 0x7faf3c147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faf2806f430 0x7faf28071940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 --2- 172.21.15.136:0/875201723 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf24004810 0x7faf3c1423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 >> 172.21.15.136:0/875201723 conn(0x7faf3c0c9d50 msgr2=0x7faf3c151650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 shutdown_connections 2024-04-25T11:33:25.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:25.617+0000 7faf42359640 1 -- 172.21.15.136:0/875201723 wait complete. 2024-04-25T11:33:25.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:27.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:27.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 Processor -- start 2024-04-25T11:33:27.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 -- start start 2024-04-25T11:33:27.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7b2c14f840 0x7f7b2c074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b2c073200 0x7f7b2c073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b2c0c3750 con 0x7f7b2c0c3c30 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7b2c06dc00 con 0x7f7b2c073200 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b309e6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7b2c14fc50 con 0x7f7b2c14f840 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b2c073200 0x7f7b2c073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7b2c073200 0x7f7b2c073610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52422/0 (socket says 172.21.15.136:52422) 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b2ad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7b2c0c3c30 0x7f7b2c0c4040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38554/0 (socket says 172.21.15.136:38554) 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 -- 172.21.15.136:0/970885977 learned_addr learned my addr 172.21.15.136:0/970885977 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 -- 172.21.15.136:0/970885977 >> v2:172.21.15.157:3300/0 conn(0x7f7b2c14f840 msgr2=0x7f7b2c074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b2a59c640 1 --2- 172.21.15.136:0/970885977 >> v2:172.21.15.157:3300/0 conn(0x7f7b2c14f840 0x7f7b2c074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 --2- 172.21.15.136:0/970885977 >> v2:172.21.15.157:3300/0 conn(0x7f7b2c14f840 0x7f7b2c074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 -- 172.21.15.136:0/970885977 >> v1:172.21.15.136:6789/0 conn(0x7f7b2c0c3c30 legacy=0x7f7b2c0c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b29d9b640 1 -- 172.21.15.136:0/970885977 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907176662 0 0) 0x7f7b2c0c3750 con 0x7f7b2c0c3c30 2024-04-25T11:33:27.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.813+0000 7f7b23fff640 1 -- 172.21.15.136:0/970885977 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c0c3190 con 0x7f7b2c073200 2024-04-25T11:33:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b23fff640 1 --2- 172.21.15.136:0/970885977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c073610 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009870 tx=0x7f7b1401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ef85f0feca6529 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b29d9b640 1 -- 172.21.15.136:0/970885977 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b140b0070 con 0x7f7b2c073200 2024-04-25T11:33:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b29d9b640 1 -- 172.21.15.136:0/970885977 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b140ab600 con 0x7f7b2c073200 2024-04-25T11:33:27.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b29d9b640 1 -- 172.21.15.136:0/970885977 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b140ab9d0 con 0x7f7b2c073200 2024-04-25T11:33:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 -- 172.21.15.136:0/970885977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 msgr2=0x7f7b2c073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 --2- 172.21.15.136:0/970885977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c073610 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009870 tx=0x7f7b1401c520 comp rx=0 tx=0).stop 2024-04-25T11:33:27.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 -- 172.21.15.136:0/970885977 shutdown_connections 2024-04-25T11:33:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 --2- 172.21.15.136:0/970885977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c073610 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 --2- 172.21.15.136:0/970885977 >> v2:172.21.15.157:3300/0 conn(0x7f7b2c14f840 0x7f7b2c074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 -- 172.21.15.136:0/970885977 >> 172.21.15.136:0/970885977 conn(0x7f7b2c0ca5a0 msgr2=0x7f7b2c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 -- 172.21.15.136:0/970885977 shutdown_connections 2024-04-25T11:33:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.817+0000 7f7b309e6640 1 -- 172.21.15.136:0/970885977 wait complete. 2024-04-25T11:33:27.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 Processor -- start 2024-04-25T11:33:27.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- start start 2024-04-25T11:33:27.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 0x7f7b2c142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b2c14f840 0x7f7b2c13f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1597f0 con 0x7f7b2c073200 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c153380 con 0x7f7b2c0c3c30 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2c159970 con 0x7f7b2c14f840 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52424/0 (socket says 172.21.15.136:52424) 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 -- 172.21.15.136:0/3561602975 learned_addr learned my addr 172.21.15.136:0/3561602975 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2ad9d640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b2c14f840 0x7f7b2c13f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b23fff640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 0x7f7b2c142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 -- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b2c14f840 msgr2=0x7f7b2c13f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b2c14f840 0x7f7b2c13f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 -- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 msgr2=0x7f7b2c142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 0x7f7b2c142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:27.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 -- 172.21.15.136:0/3561602975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c155c20 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b23fff640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 0x7f7b2c142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b2a59c640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c7f0 tx=0x7f7b1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c015070 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b1c007820 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c140f90 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c007c10 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.821+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c155ee0 con 0x7f7b2c073200 2024-04-25T11:33:27.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b2c1f6ce0 con 0x7f7b2c073200 2024-04-25T11:33:27.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7b1c018070 con 0x7f7b2c073200 2024-04-25T11:33:27.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b22ffd640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 0x7f7b0c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:27.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7251+0+0 (secure 0 0 0) 0x7f7b1c08fd00 con 0x7f7b2c073200 2024-04-25T11:33:27.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b23fff640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 0x7f7b0c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:27.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.825+0000 7f7b23fff640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 0x7f7b0c071910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7b140ad000 tx=0x7f7b14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:27.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:27.829+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7b1c05c750 con 0x7f7b2c073200 2024-04-25T11:33:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.053+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b2c06a350 con 0x7f7b2c073200 2024-04-25T11:33:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.053+0000 7f7b22ffd640 1 -- 172.21.15.136:0/3561602975 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6931 (secure 0 0 0) 0x7f7b1c05c050 con 0x7f7b2c073200 2024-04-25T11:33:28.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:28.037 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5449,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:00.184541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5036,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5036":{"gid":5036,"name":"b","rank":1,"incarnation":44,"state":"up:active","state_seq":13,"addr":"172.21.15.157:6837/3688751078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3688751078},{"type":"v1","addr":"172.21.15.157:6837","nonce":3688751078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5385,5356,5238,5517,5036]},"id":1}]} 2024-04-25T11:33:28.038 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 55 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 msgr2=0x7f7b0c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 0x7f7b0c071910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7b140ad000 tx=0x7f7b14008040 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 msgr2=0x7f7b2c1f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00c7f0 tx=0x7f7b1c00ccb0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 shutdown_connections 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7b0c06f400 0x7f7b0c071910 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b2c14f840 0x7f7b2c13f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7b2c0c3c30 0x7f7b2c142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 --2- 172.21.15.136:0/3561602975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b2c073200 0x7f7b2c1f70a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:28.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 >> 172.21.15.136:0/3561602975 conn(0x7f7b2c0ca5a0 msgr2=0x7f7b2c144770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:28.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 shutdown_connections 2024-04-25T11:33:28.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:28.057+0000 7f7b309e6640 1 -- 172.21.15.136:0/3561602975 wait complete. 2024-04-25T11:33:28.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:33:30.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:30.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.253+0000 7f3a3e72e640 1 Processor -- start 2024-04-25T11:33:30.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.253+0000 7f3a3e72e640 1 -- start start 2024-04-25T11:33:30.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3e72e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a38156330 0x7f3a3815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3e72e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a3815d010 con 0x7f3a380b5ba0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3e72e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a380ccb80 con 0x7f3a380cdee0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3e72e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a381602f0 con 0x7f3a38156330 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3cccb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3a380b5ba0 0x7f3a381669d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38558/0 (socket says 172.21.15.136:38558) 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3cccb640 1 -- 172.21.15.136:0/1033132985 learned_addr learned my addr 172.21.15.136:0/1033132985 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a37fff640 1 --2- 172.21.15.136:0/1033132985 >> v2:172.21.15.157:3300/0 conn(0x7f3a38156330 0x7f3a3815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679632449 0 0) 0x7f3a300014a0 con 0x7f3a380b5ba0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a24003530 con 0x7f3a380b5ba0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 505070696 0 0) 0x7f3a24003530 con 0x7f3a380b5ba0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 >> v2:172.21.15.157:3300/0 conn(0x7f3a38156330 msgr2=0x7f3a3815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 --2- 172.21.15.136:0/1033132985 >> v2:172.21.15.157:3300/0 conn(0x7f3a38156330 0x7f3a3815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 >> v1:172.21.15.157:6789/0 conn(0x7f3a380cdee0 legacy=0x7f3a380b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a38156740 con 0x7f3a380b5ba0 2024-04-25T11:33:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f3a30001bd0 con 0x7f3a380b5ba0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 >> v1:172.21.15.136:6789/0 conn(0x7f3a380b5ba0 legacy=0x7f3a381669d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240039c0 0x7f3a24003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 0x7f3a24004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240051d0 0x7f3a2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381602f0 con 0x7f3a24004810 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2400c160 con 0x7f3a240051d0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2400c360 con 0x7f3a240039c0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 0x7f3a24004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a37fff640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240039c0 0x7f3a24003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3cccb640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240051d0 0x7f3a2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 -- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240039c0 msgr2=0x7f3a24003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a240039c0 0x7f3a24003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 -- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240051d0 msgr2=0x7f3a2400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:30.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a240051d0 0x7f3a2400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a2ffff640 1 -- 172.21.15.136:0/1033132985 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a24003530 con 0x7f3a24004810 2024-04-25T11:33:30.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.257+0000 7f3a3cccb640 1 -- 172.21.15.136:0/1033132985 reap_dead start 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a2ffff640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 0x7f3a24004c20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3a20001750 tx=0x7f3a2001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9ddb12aabcd10c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a200013a0 con 0x7f3a24004810 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a200052f0 con 0x7f3a24004810 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a377fe640 1 -- 172.21.15.136:0/1033132985 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20003840 con 0x7f3a24004810 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 -- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 msgr2=0x7f3a24004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 0x7f3a24004c20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3a20001750 tx=0x7f3a2001c3c0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 -- 172.21.15.136:0/1033132985 shutdown_connections 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/1033132985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a24004810 0x7f3a24004c20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 -- 172.21.15.136:0/1033132985 >> 172.21.15.136:0/1033132985 conn(0x7f3a3806d6c0 msgr2=0x7f3a3806e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:30.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 -- 172.21.15.136:0/1033132985 shutdown_connections 2024-04-25T11:33:30.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.261+0000 7f3a3e72e640 1 -- 172.21.15.136:0/1033132985 wait complete. 2024-04-25T11:33:30.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 Processor -- start 2024-04-25T11:33:30.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- start start 2024-04-25T11:33:30.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a24004810 0x7f3a381f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a38159380 0x7f3a38157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3815fca0 con 0x7f3a38148810 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3815f9c0 con 0x7f3a38159380 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3815ff80 con 0x7f3a24004810 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52444/0 (socket says 172.21.15.136:52444) 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 -- 172.21.15.136:0/907885057 learned_addr learned my addr 172.21.15.136:0/907885057 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a37fff640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a24004810 0x7f3a381f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a2ffff640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a38159380 0x7f3a38157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 -- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a24004810 msgr2=0x7f3a381f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a24004810 0x7f3a381f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 -- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a38159380 msgr2=0x7f3a38157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a38159380 0x7f3a38157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 -- 172.21.15.136:0/907885057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a38159790 con 0x7f3a38148810 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3cccb640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3a3808f410 tx=0x7f3a30007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30019070 con 0x7f3a38148810 2024-04-25T11:33:30.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3000ea80 con 0x7f3a38148810 2024-04-25T11:33:30.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a381e5d80 con 0x7f3a38148810 2024-04-25T11:33:30.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000ee70 con 0x7f3a38148810 2024-04-25T11:33:30.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.265+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a381f2fa0 con 0x7f3a38148810 2024-04-25T11:33:30.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3806a350 con 0x7f3a38148810 2024-04-25T11:33:30.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3a3001c070 con 0x7f3a38148810 2024-04-25T11:33:30.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a357fa640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 0x7f3a18071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f3a3008f7a0 con 0x7f3a38148810 2024-04-25T11:33:30.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a37fff640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 0x7f3a18071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.269+0000 7f3a37fff640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 0x7f3a18071940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009ab0 tx=0x7f3a28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.273+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3a30023020 con 0x7f3a38148810 2024-04-25T11:33:30.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.485+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a380cd9c0 con 0x7f3a38148810 2024-04-25T11:33:30.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a357fa640 1 -- 172.21.15.136:0/907885057 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6098 (secure 0 0 0) 0x7f3a3815d010 con 0x7f3a38148810 2024-04-25T11:33:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:30.472 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:28.924315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5517]},"id":1}]} 2024-04-25T11:33:30.472 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 56 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 msgr2=0x7f3a18071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 0x7f3a18071940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009ab0 tx=0x7f3a28008040 comp rx=0 tx=0).stop 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 msgr2=0x7f3a38156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3a3808f410 tx=0x7f3a30007bc0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 shutdown_connections 2024-04-25T11:33:30.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a38148810 0x7f3a38156960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a38159380 0x7f3a38157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3a1806f430 0x7f3a18071940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 --2- 172.21.15.136:0/907885057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a24004810 0x7f3a381f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 >> 172.21.15.136:0/907885057 conn(0x7f3a3806d6c0 msgr2=0x7f3a38153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 shutdown_connections 2024-04-25T11:33:30.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.489+0000 7f3a3e72e640 1 -- 172.21.15.136:0/907885057 wait complete. 2024-04-25T11:33:30.497 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-25T11:33:30.497 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:33:30.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:30.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.649+0000 7effc736f640 1 Processor -- start 2024-04-25T11:33:30.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.649+0000 7effc736f640 1 -- start start 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc736f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00ca290 0x7effc00ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc736f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7effc015a430 0x7effc015a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc736f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effc015d750 con 0x7effc00b5060 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc736f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7effc015da30 con 0x7effc015a430 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc736f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effc015dd10 con 0x7effc00ca290 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7effc015a430 0x7effc015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7effc015a430 0x7effc015a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52456/0 (socket says 172.21.15.136:52456) 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 -- 172.21.15.136:0/808226028 learned_addr learned my addr 172.21.15.136:0/808226028 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc490a640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00ca290 0x7effc00ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 -- 172.21.15.136:0/808226028 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00ca290 msgr2=0x7effc00ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:30.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00ca290 0x7effc00ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 -- 172.21.15.136:0/808226028 >> v1:172.21.15.157:6789/0 conn(0x7effc00b5060 legacy=0x7effc00c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:30.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.653+0000 7effc590c640 1 -- 172.21.15.136:0/808226028 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc015dff0 con 0x7effc015a430 2024-04-25T11:33:30.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc590c640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc015a430 0x7effc015a840 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e50 tx=0x7effb801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cbf9223d72a1024 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effb7fff640 1 -- 172.21.15.136:0/808226028 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb80b4070 con 0x7effc015a430 2024-04-25T11:33:30.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effb7fff640 1 -- 172.21.15.136:0/808226028 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb80af400 con 0x7effc015a430 2024-04-25T11:33:30.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effb7fff640 1 -- 172.21.15.136:0/808226028 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb80af7d0 con 0x7effc015a430 2024-04-25T11:33:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- 172.21.15.136:0/808226028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc015a430 msgr2=0x7effc015a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc015a430 0x7effc015a840 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7effb8009e50 tx=0x7effb801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- 172.21.15.136:0/808226028 shutdown_connections 2024-04-25T11:33:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc015a430 0x7effc015a840 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 --2- 172.21.15.136:0/808226028 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00ca290 0x7effc00ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- 172.21.15.136:0/808226028 >> 172.21.15.136:0/808226028 conn(0x7effc0070f70 msgr2=0x7effc0068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:30.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- 172.21.15.136:0/808226028 shutdown_connections 2024-04-25T11:33:30.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- 172.21.15.136:0/808226028 wait complete. 2024-04-25T11:33:30.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 Processor -- start 2024-04-25T11:33:30.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.657+0000 7effc736f640 1 -- start start 2024-04-25T11:33:30.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 0x7effc00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effc015a430 0x7effc01f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7effc015ad80 con 0x7effc00ca290 2024-04-25T11:33:30.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7effc0159f10 con 0x7effc015a430 2024-04-25T11:33:30.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effc015d3f0 con 0x7effc00b5060 2024-04-25T11:33:30.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52470/0 (socket says 172.21.15.136:52470) 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 -- 172.21.15.136:0/3682671258 learned_addr learned my addr 172.21.15.136:0/3682671258 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc510b640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 0x7effc00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc590c640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effc015a430 0x7effc01f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 -- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 msgr2=0x7effc00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 0x7effc00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 -- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effc015a430 msgr2=0x7effc01f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effc015a430 0x7effc01f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 -- 172.21.15.136:0/3682671258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc00cebb0 con 0x7effc00ca290 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc510b640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 0x7effc00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc490a640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7effb000d650 tx=0x7effb000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000b6c0 con 0x7effc00ca290 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb0019070 con 0x7effc00ca290 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc01462b0 con 0x7effc00ca290 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000ba80 con 0x7effc00ca290 2024-04-25T11:33:30.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc01470b0 con 0x7effc00ca290 2024-04-25T11:33:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc01f2770 con 0x7effc00ca290 2024-04-25T11:33:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7effb000bc20 con 0x7effc00ca290 2024-04-25T11:33:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 0x7effa4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.661+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7effb008fc40 con 0x7effc00ca290 2024-04-25T11:33:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.665+0000 7effc510b640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 0x7effa4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:30.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.665+0000 7effc510b640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 0x7effa4071940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7effbc00a850 tx=0x7effbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:30.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.669+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7effb005c5f0 con 0x7effc00ca290 2024-04-25T11:33:30.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effc006a350 con 0x7effc00ca290 2024-04-25T11:33:30.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effb5ffb640 1 -- 172.21.15.136:0/3682671258 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6098 (secure 0 0 0) 0x7effb005bef0 con 0x7effc00ca290 2024-04-25T11:33:30.893 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:30.893 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:28.924315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5517]},"id":1}]} 2024-04-25T11:33:30.893 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 56 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 msgr2=0x7effa4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 0x7effa4071940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7effbc00a850 tx=0x7effbc008040 comp rx=0 tx=0).stop 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 msgr2=0x7effc01505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7effb000d650 tx=0x7effb000db10 comp rx=0 tx=0).stop 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 shutdown_connections 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effc015a430 0x7effc01f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effc00ca290 0x7effc01505d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effa406f430 0x7effa4071940 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 --2- 172.21.15.136:0/3682671258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effc00b5060 0x7effc00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:30.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 >> 172.21.15.136:0/3682671258 conn(0x7effc0070f70 msgr2=0x7effc00c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:30.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 shutdown_connections 2024-04-25T11:33:30.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:30.909+0000 7effc736f640 1 -- 172.21.15.136:0/3682671258 wait complete. 2024-04-25T11:33:30.915 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:33:28.924315+0000", "qdb_cluster": [ 5238, 5356, 5385, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:33:32.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:33.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.069+0000 7f1f974de640 1 Processor -- start 2024-04-25T11:33:33.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.069+0000 7f1f974de640 1 -- start start 2024-04-25T11:33:33.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.069+0000 7f1f974de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f900c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1f9015a3d0 0x7f1f9015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f900c9af0 con 0x7f1f900c96e0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1f9006cc10 con 0x7f1f9015a3d0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1f900cb220 con 0x7f1f900c79e0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1f9015a3d0 0x7f1f9015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1f9015a3d0 0x7f1f9015a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52482/0 (socket says 172.21.15.136:52482) 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 -- 172.21.15.136:0/1345309870 learned_addr learned my addr 172.21.15.136:0/1345309870 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f94a79640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f900c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 -- 172.21.15.136:0/1345309870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 msgr2=0x7f1f900c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f900c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 -- 172.21.15.136:0/1345309870 >> v1:172.21.15.136:6789/0 conn(0x7f1f900c96e0 legacy=0x7f1f901540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 -- 172.21.15.136:0/1345309870 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f900cb500 con 0x7f1f9015a3d0 2024-04-25T11:33:33.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f87fff640 1 -- 172.21.15.136:0/1345309870 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2931281958 0 0) 0x7f1f900c9af0 con 0x7f1f900c96e0 2024-04-25T11:33:33.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f95a7b640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f9015a3d0 0x7f1f9015a7e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009e50 tx=0x7f1f8801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50eb3dc9c4d3e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:33.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f87fff640 1 -- 172.21.15.136:0/1345309870 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f880a5700 con 0x7f1f9015a3d0 2024-04-25T11:33:33.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f87fff640 1 -- 172.21.15.136:0/1345309870 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f880a58a0 con 0x7f1f9015a3d0 2024-04-25T11:33:33.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f87fff640 1 -- 172.21.15.136:0/1345309870 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f880a5c70 con 0x7f1f9015a3d0 2024-04-25T11:33:33.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- 172.21.15.136:0/1345309870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f9015a3d0 msgr2=0x7f1f9015a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f9015a3d0 0x7f1f9015a7e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009e50 tx=0x7f1f8801ca30 comp rx=0 tx=0).stop 2024-04-25T11:33:33.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- 172.21.15.136:0/1345309870 shutdown_connections 2024-04-25T11:33:33.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f9015a3d0 0x7f1f9015a7e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 --2- 172.21.15.136:0/1345309870 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f900c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.073+0000 7f1f974de640 1 -- 172.21.15.136:0/1345309870 >> 172.21.15.136:0/1345309870 conn(0x7f1f9006f9d0 msgr2=0x7f1f9006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:33.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- 172.21.15.136:0/1345309870 shutdown_connections 2024-04-25T11:33:33.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- 172.21.15.136:0/1345309870 wait complete. 2024-04-25T11:33:33.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 Processor -- start 2024-04-25T11:33:33.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- start start 2024-04-25T11:33:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f901f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 0x7f1f900cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1f900ced40 con 0x7f1f900c96e0 2024-04-25T11:33:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9006d440 con 0x7f1f9015a3d0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.077+0000 7f1f974de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1f90157e80 con 0x7f1f900c79e0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52498/0 (socket says 172.21.15.136:52498) 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 -- 172.21.15.136:0/2041481878 learned_addr learned my addr 172.21.15.136:0/2041481878 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f9527a640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f901f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f95a7b640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 0x7f1f900cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 -- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 msgr2=0x7f1f901f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f901f3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 -- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 msgr2=0x7f1f900cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 0x7f1f900cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 -- 172.21.15.136:0/2041481878 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f901f4000 con 0x7f1f900c96e0 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f95a7b640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 0x7f1f900cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f9527a640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f901f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f94a79640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000d620 tx=0x7f1f8000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:33.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f800125f0 con 0x7f1f900c96e0 2024-04-25T11:33:33.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f80019070 con 0x7f1f900c96e0 2024-04-25T11:33:33.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f90159940 con 0x7f1f900c96e0 2024-04-25T11:33:33.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f800129b0 con 0x7f1f900c96e0 2024-04-25T11:33:33.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f901f32c0 con 0x7f1f900c96e0 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f901f30b0 con 0x7f1f900c96e0 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1f80012b50 con 0x7f1f900c96e0 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 0x7f1f74071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f1f8008fcc0 con 0x7f1f900c96e0 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.081+0000 7f1f9527a640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 0x7f1f74071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:33.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.085+0000 7f1f9527a640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 0x7f1f74071940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000f70 tx=0x7f1f8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:33.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.089+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1f8005c5e0 con 0x7f1f900c96e0 2024-04-25T11:33:33.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.317+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f9006a350 con 0x7f1f900c96e0 2024-04-25T11:33:33.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f85ffb640 1 -- 172.21.15.136:0/2041481878 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6098 (secure 0 0 0) 0x7f1f8005bee0 con 0x7f1f900c96e0 2024-04-25T11:33:33.303 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:33.304 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:28.924315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5517]},"id":1}]} 2024-04-25T11:33:33.304 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 56 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 msgr2=0x7f1f74071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 0x7f1f74071940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c000f70 tx=0x7f1f8c009040 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 msgr2=0x7f1f90142910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000d620 tx=0x7f1f8000dae0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 shutdown_connections 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f9015a3d0 0x7f1f900cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f900c96e0 0x7f1f90142910 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1f7406f430 0x7f1f74071940 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 --2- 172.21.15.136:0/2041481878 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f900c79e0 0x7f1f901f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:33.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 >> 172.21.15.136:0/2041481878 conn(0x7f1f9006f9d0 msgr2=0x7f1f90153070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:33.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 shutdown_connections 2024-04-25T11:33:33.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:33.321+0000 7f1f974de640 1 -- 172.21.15.136:0/2041481878 wait complete. 2024-04-25T11:33:33.328 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:35.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:35.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.525+0000 7fce7dc39640 1 Processor -- start 2024-04-25T11:33:35.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.525+0000 7fce7dc39640 1 -- start start 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce7dc39640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce7814f840 0x7fce78074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce7dc39640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce78073610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce7dc39640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce780c3750 con 0x7fce780c3c30 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce7dc39640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fce7806dc00 con 0x7fce7814f840 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce7dc39640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce7814fc50 con 0x7fce78073200 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce7814f840 0x7fce78074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce7814f840 0x7fce78074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39266/0 (socket says 172.21.15.136:39266) 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 -- 172.21.15.136:0/882583363 learned_addr learned my addr 172.21.15.136:0/882583363 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 -- 172.21.15.136:0/882583363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 msgr2=0x7fce78073610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce76ffd640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce78073610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce78073610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 -- 172.21.15.136:0/882583363 >> v1:172.21.15.157:6789/0 conn(0x7fce780c3c30 legacy=0x7fce780c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:35.512 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 -- 172.21.15.136:0/882583363 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce780c3190 con 0x7fce7814f840 2024-04-25T11:33:35.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce777fe640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce78074980 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fce68009870 tx=0x7fce6801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5148123f6061b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:35.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.529+0000 7fce767fc640 1 -- 172.21.15.136:0/882583363 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680b0070 con 0x7fce7814f840 2024-04-25T11:33:35.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce767fc640 1 -- 172.21.15.136:0/882583363 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce680ab600 con 0x7fce7814f840 2024-04-25T11:33:35.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce767fc640 1 -- 172.21.15.136:0/882583363 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce680ab9d0 con 0x7fce7814f840 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- 172.21.15.136:0/882583363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 msgr2=0x7fce78074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce78074980 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fce68009870 tx=0x7fce6801c520 comp rx=0 tx=0).stop 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- 172.21.15.136:0/882583363 shutdown_connections 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce78073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 --2- 172.21.15.136:0/882583363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce78074980 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- 172.21.15.136:0/882583363 >> 172.21.15.136:0/882583363 conn(0x7fce780ca5a0 msgr2=0x7fce780ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- 172.21.15.136:0/882583363 shutdown_connections 2024-04-25T11:33:35.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- 172.21.15.136:0/882583363 wait complete. 2024-04-25T11:33:35.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 Processor -- start 2024-04-25T11:33:35.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.533+0000 7fce7dc39640 1 -- start start 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce781436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce780c3c30 0x7fce7813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce781597f0 con 0x7fce7814f840 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce78153380 con 0x7fce780c3c30 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce78159970 con 0x7fce78073200 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39282/0 (socket says 172.21.15.136:39282) 2024-04-25T11:33:35.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 -- 172.21.15.136:0/3833508403 learned_addr learned my addr 172.21.15.136:0/3833508403 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce777fe640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce781436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce76ffd640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce780c3c30 0x7fce7813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 -- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 msgr2=0x7fce781436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce781436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 -- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce780c3c30 msgr2=0x7fce7813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce780c3c30 0x7fce7813e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 -- 172.21.15.136:0/3833508403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce78143be0 con 0x7fce7814f840 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce77fff640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fce7000ba10 tx=0x7fce7000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70019070 con 0x7fce7814f840 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce7000cb40 con 0x7fce7814f840 2024-04-25T11:33:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70014440 con 0x7fce7814f840 2024-04-25T11:33:35.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce781580c0 con 0x7fce7814f840 2024-04-25T11:33:35.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce78158680 con 0x7fce7814f840 2024-04-25T11:33:35.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.537+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce7806a350 con 0x7fce7814f840 2024-04-25T11:33:35.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.541+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fce7001c070 con 0x7fce7814f840 2024-04-25T11:33:35.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.541+0000 7fce5ffff640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 0x7fce64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:35.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.541+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fce7008f3a0 con 0x7fce7814f840 2024-04-25T11:33:35.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.541+0000 7fce777fe640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 0x7fce64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:35.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.541+0000 7fce777fe640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 0x7fce64071940 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fce680ad000 tx=0x7fce68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.549+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce7005bd50 con 0x7fce7814f840 2024-04-25T11:33:35.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.781+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce780c9430 con 0x7fce7814f840 2024-04-25T11:33:35.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.781+0000 7fce5ffff640 1 -- 172.21.15.136:0/3833508403 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6099 (secure 0 0 0) 0x7fce7005b650 con 0x7fce7814f840 2024-04-25T11:33:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:35.767 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:33.939098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:resolve","state_seq":9,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5385,5356,5238]},"id":1}]} 2024-04-25T11:33:35.767 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 57 2024-04-25T11:33:35.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 msgr2=0x7fce64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:35.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 0x7fce64071940 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fce680ad000 tx=0x7fce68008040 comp rx=0 tx=0).stop 2024-04-25T11:33:35.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 msgr2=0x7fce7813eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fce7000ba10 tx=0x7fce7000bed0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 shutdown_connections 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce7814f840 0x7fce7813eca0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce780c3c30 0x7fce7813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fce6406f430 0x7fce64071940 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 --2- 172.21.15.136:0/3833508403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce78073200 0x7fce781436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 >> 172.21.15.136:0/3833508403 conn(0x7fce780ca5a0 msgr2=0x7fce78075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 shutdown_connections 2024-04-25T11:33:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:35.785+0000 7fce7dc39640 1 -- 172.21.15.136:0/3833508403 wait complete. 2024-04-25T11:33:35.792 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:37.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:37.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.981+0000 7f279221f640 1 Processor -- start 2024-04-25T11:33:37.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.981+0000 7f279221f640 1 -- start start 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f279221f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f278c15c070 0x7f278c06d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f279221f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f278c15c480 0x7f278c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f279221f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278c0b4480 con 0x7f278c0b5810 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f279221f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f278c0b4a90 con 0x7f278c15c480 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f279221f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f278c15c990 con 0x7f278c15c070 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f278c15c480 0x7f278c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f278c0b5810 0x7f278c0b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52408/0 (socket says 172.21.15.136:52408) 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f278c15c480 0x7f278c06da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39298/0 (socket says 172.21.15.136:39298) 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278b7fe640 1 -- 172.21.15.136:0/2960828369 learned_addr learned my addr 172.21.15.136:0/2960828369 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:37.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f2783fff640 1 --2- 172.21.15.136:0/2960828369 >> v2:172.21.15.157:3300/0 conn(0x7f278c15c070 0x7f278c06d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 -- 172.21.15.136:0/2960828369 >> v2:172.21.15.157:3300/0 conn(0x7f278c15c070 msgr2=0x7f278c06d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:37.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 --2- 172.21.15.136:0/2960828369 >> v2:172.21.15.157:3300/0 conn(0x7f278c15c070 0x7f278c06d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:37.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 -- 172.21.15.136:0/2960828369 >> v1:172.21.15.136:6789/0 conn(0x7f278c0b5810 legacy=0x7f278c0b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:37.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 -- 172.21.15.136:0/2960828369 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c15cc70 con 0x7f278c15c480 2024-04-25T11:33:37.969 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278affd640 1 -- 172.21.15.136:0/2960828369 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146086799 0 0) 0x7f278c0b4480 con 0x7f278c0b5810 2024-04-25T11:33:37.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.985+0000 7f278bfff640 1 --2- 172.21.15.136:0/2960828369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c06da50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f278c069490 tx=0x7f278401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9eceb033ae2baf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:37.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f278affd640 1 -- 172.21.15.136:0/2960828369 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840b4070 con 0x7f278c15c480 2024-04-25T11:33:37.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f278affd640 1 -- 172.21.15.136:0/2960828369 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27840af600 con 0x7f278c15c480 2024-04-25T11:33:37.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f278affd640 1 -- 172.21.15.136:0/2960828369 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840af9d0 con 0x7f278c15c480 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- 172.21.15.136:0/2960828369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 msgr2=0x7f278c06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 --2- 172.21.15.136:0/2960828369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c06da50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f278c069490 tx=0x7f278401f110 comp rx=0 tx=0).stop 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- 172.21.15.136:0/2960828369 shutdown_connections 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 --2- 172.21.15.136:0/2960828369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c06da50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 --2- 172.21.15.136:0/2960828369 >> v2:172.21.15.157:3300/0 conn(0x7f278c15c070 0x7f278c06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:37.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- 172.21.15.136:0/2960828369 >> 172.21.15.136:0/2960828369 conn(0x7f278c0743a0 msgr2=0x7f278c0d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:37.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- 172.21.15.136:0/2960828369 shutdown_connections 2024-04-25T11:33:37.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- 172.21.15.136:0/2960828369 wait complete. 2024-04-25T11:33:37.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 Processor -- start 2024-04-25T11:33:37.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.989+0000 7f279221f640 1 -- start start 2024-04-25T11:33:37.976 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 0x7f278c0c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 0x7f278c1f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f278c06e030 con 0x7f278c15c480 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f278c15cf50 con 0x7f278c0b5810 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f278c0cbee0 con 0x7f278c15c070 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39312/0 (socket says 172.21.15.136:39312) 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 -- 172.21.15.136:0/405461667 learned_addr learned my addr 172.21.15.136:0/405461667 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278b7fe640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 0x7f278c0c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2783fff640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 0x7f278c1f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 -- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 msgr2=0x7f278c1f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 0x7f278c1f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:37.977 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 -- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 msgr2=0x7f278c0c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 0x7f278c0c59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 -- 172.21.15.136:0/405461667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c0c4f30 con 0x7f278c15c480 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2783fff640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 0x7f278c1f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278b7fe640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 0x7f278c0c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f278bfff640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f278401cd40 tx=0x7f278401beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840b4040 con 0x7f278c15c480 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27840c1070 con 0x7f278c15c480 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278c0c7a80 con 0x7f278c15c480 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784009450 con 0x7f278c15c480 2024-04-25T11:33:37.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c0c51f0 con 0x7f278c15c480 2024-04-25T11:33:37.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c0c77f0 con 0x7f278c15c480 2024-04-25T11:33:37.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f27840ba6b0 con 0x7f278c15c480 2024-04-25T11:33:37.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 0x7f276c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:37.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.993+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f278412e300 con 0x7f278c15c480 2024-04-25T11:33:37.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.997+0000 7f278b7fe640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 0x7f276c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:37.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:37.997+0000 7f278b7fe640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 0x7f276c071910 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f277c000f70 tx=0x7f277c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:37.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.001+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2784018c70 con 0x7f278c15c480 2024-04-25T11:33:38.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f278c06a350 con 0x7f278c15c480 2024-04-25T11:33:38.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f2788ff9640 1 -- 172.21.15.136:0/405461667 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6099 (secure 0 0 0) 0x7f27840fac20 con 0x7f278c15c480 2024-04-25T11:33:38.224 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:38.224 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:33.939098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:resolve","state_seq":9,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5385,5356,5238]},"id":1}]} 2024-04-25T11:33:38.224 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 57 2024-04-25T11:33:38.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 msgr2=0x7f276c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:38.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 0x7f276c071910 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f277c000f70 tx=0x7f277c009040 comp rx=0 tx=0).stop 2024-04-25T11:33:38.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 msgr2=0x7f278c0c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:38.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f278401cd40 tx=0x7f278401beb0 comp rx=0 tx=0).stop 2024-04-25T11:33:38.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 shutdown_connections 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f278c15c480 0x7f278c0c8fc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f276c06f400 0x7f276c071910 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f278c15c070 0x7f278c1f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 --2- 172.21.15.136:0/405461667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f278c0b5810 0x7f278c0c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 >> 172.21.15.136:0/405461667 conn(0x7f278c0743a0 msgr2=0x7f278c0ce6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 shutdown_connections 2024-04-25T11:33:38.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:38.241+0000 7f279221f640 1 -- 172.21.15.136:0/405461667 wait complete. 2024-04-25T11:33:38.253 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:40.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:40.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 Processor -- start 2024-04-25T11:33:40.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 -- start start 2024-04-25T11:33:40.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa8c40ca290 0x7fa8c40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8c415a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c415d750 con 0x7fa8c40b5060 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c415da30 con 0x7fa8c40ca290 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c888b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c415dd10 con 0x7fa8c415a430 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c259c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa8c40b5060 0x7fa8c40c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52410/0 (socket says 172.21.15.136:52410) 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c259c640 1 -- 172.21.15.136:0/1536502463 learned_addr learned my addr 172.21.15.136:0/1536502463 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c2d9d640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8c415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/1536502463 >> v2:172.21.15.157:3300/0 conn(0x7fa8c40ca290 0x7fa8c40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302384341 0 0) 0x7fa8c415d750 con 0x7fa8c40b5060 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b0003530 con 0x7fa8c40b5060 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926540266 0 0) 0x7fa8b0003530 con 0x7fa8c40b5060 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 msgr2=0x7fa8c415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8c415a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 >> v2:172.21.15.157:3300/0 conn(0x7fa8c40ca290 msgr2=0x7fa8c40ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 --2- 172.21.15.136:0/1536502463 >> v2:172.21.15.157:3300/0 conn(0x7fa8c40ca290 0x7fa8c40ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c415dff0 con 0x7fa8c40b5060 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fa8b4001680 con 0x7fa8c40b5060 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 >> v1:172.21.15.136:6789/0 conn(0x7fa8c40b5060 legacy=0x7fa8c40c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8b00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b0004d50 0x7fa8b0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c415da30 con 0x7fa8b0003f40 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c415dd10 con 0x7fa8b0004d50 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b000bab0 con 0x7fa8c415a430 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c259c640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8b00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c2d9d640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b0004d50 0x7fa8b0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 -- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 msgr2=0x7fa8b00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8b00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 -- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b0004d50 msgr2=0x7fa8b0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:40.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b0004d50 0x7fa8b0005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8bbfff640 1 -- 172.21.15.136:0/1536502463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b0003530 con 0x7fa8b0003f40 2024-04-25T11:33:40.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c259c640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c415a430 0x7fa8b00039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:33:40.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.425+0000 7fa8c2d9d640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8b0004d50 0x7fa8b0005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8b0004810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009a90 tx=0x7fa8ac01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77ce42112da31a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0b2070 con 0x7fa8b0003f40 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ac0a5660 con 0x7fa8b0003f40 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c1d9b640 1 -- 172.21.15.136:0/1536502463 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac009690 con 0x7fa8b0003f40 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 -- 172.21.15.136:0/1536502463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 msgr2=0x7fa8b0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 --2- 172.21.15.136:0/1536502463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8b0004810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009a90 tx=0x7fa8ac01ca70 comp rx=0 tx=0).stop 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c2d9d640 1 -- 172.21.15.136:0/1536502463 reap_dead start 2024-04-25T11:33:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 -- 172.21.15.136:0/1536502463 shutdown_connections 2024-04-25T11:33:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 -- 172.21.15.136:0/1536502463 >> 172.21.15.136:0/1536502463 conn(0x7fa8c4070f70 msgr2=0x7fa8c4068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 -- 172.21.15.136:0/1536502463 shutdown_connections 2024-04-25T11:33:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.429+0000 7fa8c888b640 1 -- 172.21.15.136:0/1536502463 wait complete. 2024-04-25T11:33:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 Processor -- start 2024-04-25T11:33:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- start start 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8c414f4c0 0x7fa8c414f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c4150040 0x7fa8c4150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c415d3f0 con 0x7fa8b0003f40 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c4159f10 con 0x7fa8c414f4c0 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c415a840 con 0x7fa8c4150040 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39328/0 (socket says 172.21.15.136:39328) 2024-04-25T11:33:40.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 -- 172.21.15.136:0/2260277869 learned_addr learned my addr 172.21.15.136:0/2260277869 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c2d9d640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c4150040 0x7fa8c4150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8c414f4c0 0x7fa8c414f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 -- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c4150040 msgr2=0x7fa8c4150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c4150040 0x7fa8c4150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 -- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8c414f4c0 msgr2=0x7fa8c414f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8c414f4c0 0x7fa8c414f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 -- 172.21.15.136:0/2260277869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c414e560 con 0x7fa8b0003f40 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c259c640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e7d0 tx=0x7fa8b400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400c7f0 con 0x7fa8b0003f40 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b4018070 con 0x7fa8b0003f40 2024-04-25T11:33:40.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400cb90 con 0x7fa8b0003f40 2024-04-25T11:33:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c41f2410 con 0x7fa8b0003f40 2024-04-25T11:33:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.433+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c41f29b0 con 0x7fa8b0003f40 2024-04-25T11:33:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c406a350 con 0x7fa8b0003f40 2024-04-25T11:33:40.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa8b401d460 con 0x7fa8b0003f40 2024-04-25T11:33:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8baffd640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fa8b408f7e0 con 0x7fa8b0003f40 2024-04-25T11:33:40.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:40.422 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.437+0000 7fa8bbfff640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0af000 tx=0x7fa8ac0046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:40.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.445+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa8b405c070 con 0x7fa8b0003f40 2024-04-25T11:33:40.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.677+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8c40cadd0 con 0x7fa8b0003f40 2024-04-25T11:33:40.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.677+0000 7fa8baffd640 1 -- 172.21.15.136:0/2260277869 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6102 (secure 0 0 0) 0x7fa8b40640f0 con 0x7fa8b0003f40 2024-04-25T11:33:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:40.663 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:39.204929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5517]},"id":1}]} 2024-04-25T11:33:40.663 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 58 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 msgr2=0x7fa8a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0af000 tx=0x7fa8ac0046d0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 msgr2=0x7fa8c414e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400e7d0 tx=0x7fa8b400ec90 comp rx=0 tx=0).stop 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 shutdown_connections 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8a406f400 0x7fa8a4071910 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa8c4150040 0x7fa8c4150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8c414f4c0 0x7fa8c414f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 --2- 172.21.15.136:0/2260277869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b0003f40 0x7fa8c414e020 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 >> 172.21.15.136:0/2260277869 conn(0x7fa8c4070f70 msgr2=0x7fa8c40b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 shutdown_connections 2024-04-25T11:33:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:40.681+0000 7fa8c888b640 1 -- 172.21.15.136:0/2260277869 wait complete. 2024-04-25T11:33:40.686 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:42.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 Processor -- start 2024-04-25T11:33:42.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 -- start start 2024-04-25T11:33:42.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41340cb090 0x7f41340cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4134156550 con 0x7f4134073a10 2024-04-25T11:33:42.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f41340c9710 con 0x7f41340cb090 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413ac51640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4134160070 con 0x7f41340742f0 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.873+0000 7f413a450640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41340cb090 0x7f41340cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41340cb090 0x7f41340cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39332/0 (socket says 172.21.15.136:39332) 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 -- 172.21.15.136:0/3891535611 learned_addr learned my addr 172.21.15.136:0/3891535611 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413944e640 1 --1- 172.21.15.136:0/3891535611 >> v1:172.21.15.136:6789/0 conn(0x7f4134073a10 0x7f41340c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52424/0 (socket says 172.21.15.136:52424) 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f4139c4f640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 -- 172.21.15.136:0/3891535611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 msgr2=0x7f4134156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 -- 172.21.15.136:0/3891535611 >> v1:172.21.15.136:6789/0 conn(0x7f4134073a10 legacy=0x7f41340c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 -- 172.21.15.136:0/3891535611 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413415ccb0 con 0x7f41340cb090 2024-04-25T11:33:42.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f4138c4d640 1 -- 172.21.15.136:0/3891535611 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817152292 0 0) 0x7f4134156550 con 0x7f4134073a10 2024-04-25T11:33:42.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413a450640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41340cb090 0x7f41340cb4a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f412c009e50 tx=0x7f412c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983a0d473595d4dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f4138c4d640 1 -- 172.21.15.136:0/3891535611 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0a5810 con 0x7f41340cb090 2024-04-25T11:33:42.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f4138c4d640 1 -- 172.21.15.136:0/3891535611 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412c0a59b0 con 0x7f41340cb090 2024-04-25T11:33:42.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f4138c4d640 1 -- 172.21.15.136:0/3891535611 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0a5d80 con 0x7f41340cb090 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 -- 172.21.15.136:0/3891535611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41340cb090 msgr2=0x7f41340cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41340cb090 0x7f41340cb4a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f412c009e50 tx=0x7f412c01ca30 comp rx=0 tx=0).stop 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 -- 172.21.15.136:0/3891535611 shutdown_connections 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41340cb090 0x7f41340cb4a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 --2- 172.21.15.136:0/3891535611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.877+0000 7f413ac51640 1 -- 172.21.15.136:0/3891535611 >> 172.21.15.136:0/3891535611 conn(0x7f413406d950 msgr2=0x7f413406dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- 172.21.15.136:0/3891535611 shutdown_connections 2024-04-25T11:33:42.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- 172.21.15.136:0/3891535611 wait complete. 2024-04-25T11:33:42.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 Processor -- start 2024-04-25T11:33:42.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- start start 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41340cb090 0x7f41341f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f413415cf90 con 0x7f4134073a10 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4134160500 con 0x7f41340cb090 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413ac51640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4134169250 con 0x7f41340742f0 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f4139c4f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f4139c4f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39346/0 (socket says 172.21.15.136:39346) 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f4139c4f640 1 -- 172.21.15.136:0/1298424411 learned_addr learned my addr 172.21.15.136:0/1298424411 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413944e640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f413a450640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41340cb090 0x7f41341f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f4139c4f640 1 -- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 msgr2=0x7f4134146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.881+0000 7f4139c4f640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f4139c4f640 1 -- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41340cb090 msgr2=0x7f41341f8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f4139c4f640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41340cb090 0x7f41341f8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f4139c4f640 1 -- 172.21.15.136:0/1298424411 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413415d270 con 0x7f4134073a10 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f4139c4f640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f413000d620 tx=0x7f413000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413000b620 con 0x7f4134073a10 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4130019070 con 0x7f4134073a10 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41341f6320 con 0x7f4134073a10 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413000b9e0 con 0x7f4134073a10 2024-04-25T11:33:42.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4134159620 con 0x7f4134073a10 2024-04-25T11:33:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413415d440 con 0x7f4134073a10 2024-04-25T11:33:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f413000bb80 con 0x7f4134073a10 2024-04-25T11:33:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 0x7f4118071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f413008fd40 con 0x7f4134073a10 2024-04-25T11:33:42.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.885+0000 7f413944e640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 0x7f4118071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:42.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.889+0000 7f413944e640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 0x7f4118071910 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4134075d10 tx=0x7f4124007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:42.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:42.893+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f413005c660 con 0x7f4134073a10 2024-04-25T11:33:43.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.101+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f413406a2e0 con 0x7f4134073a10 2024-04-25T11:33:43.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.101+0000 7f412a7fc640 1 -- 172.21.15.136:0/1298424411 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6102 (secure 0 0 0) 0x7f413005bf60 con 0x7f4134073a10 2024-04-25T11:33:43.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:43.087 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:39.204929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5517]},"id":1}]} 2024-04-25T11:33:43.087 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 58 2024-04-25T11:33:43.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 msgr2=0x7f4118071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 0x7f4118071910 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4134075d10 tx=0x7f4124007c10 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 msgr2=0x7f41341f5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f413000d620 tx=0x7f413000dae0 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 shutdown_connections 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41340cb090 0x7f41341f8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f411806f400 0x7f4118071910 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41340742f0 0x7f4134146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 --2- 172.21.15.136:0/1298424411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4134073a10 0x7f41341f5de0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:43.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 >> 172.21.15.136:0/1298424411 conn(0x7f413406d950 msgr2=0x7f4134156010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:43.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 shutdown_connections 2024-04-25T11:33:43.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:43.105+0000 7f413ac51640 1 -- 172.21.15.136:0/1298424411 wait complete. 2024-04-25T11:33:43.109 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5238": { "addr": "172.21.15.136:6835/3406304193", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3406304193, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3406304193, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5238, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:reconnect", "state_seq": 11 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:33:39.204929+0000", "qdb_cluster": [ 5238, 5356, 5385, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 5238 } } } ], "standbys": [] } 2024-04-25T11:33:45.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:45.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.297+0000 7fe32bbe2640 1 Processor -- start 2024-04-25T11:33:45.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.297+0000 7fe32bbe2640 1 -- start start 2024-04-25T11:33:45.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe3240ca290 0x7fe3240ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe32415a430 0x7fe32415a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe32415d750 con 0x7fe3240b5060 2024-04-25T11:33:45.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe32415da30 con 0x7fe32415a430 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe32415dd10 con 0x7fe3240ca290 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe32415a430 0x7fe32415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe32415a430 0x7fe32415a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60348/0 (socket says 172.21.15.136:60348) 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 -- 172.21.15.136:0/216235927 learned_addr learned my addr 172.21.15.136:0/216235927 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32917d640 1 --2- 172.21.15.136:0/216235927 >> v2:172.21.15.157:3300/0 conn(0x7fe3240ca290 0x7fe3240ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 -- 172.21.15.136:0/216235927 >> v2:172.21.15.157:3300/0 conn(0x7fe3240ca290 msgr2=0x7fe3240ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 --2- 172.21.15.136:0/216235927 >> v2:172.21.15.157:3300/0 conn(0x7fe3240ca290 0x7fe3240ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 -- 172.21.15.136:0/216235927 >> v1:172.21.15.157:6789/0 conn(0x7fe3240b5060 legacy=0x7fe3240c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 -- 172.21.15.136:0/216235927 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32415dff0 con 0x7fe32415a430 2024-04-25T11:33:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32a17f640 1 --2- 172.21.15.136:0/216235927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32415a430 0x7fe32415a840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe3240d00d0 tx=0x7fe31c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902dc47998c7570e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32897c640 1 -- 172.21.15.136:0/216235927 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0b4070 con 0x7fe32415a430 2024-04-25T11:33:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32897c640 1 -- 172.21.15.136:0/216235927 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c0af400 con 0x7fe32415a430 2024-04-25T11:33:45.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32897c640 1 -- 172.21.15.136:0/216235927 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0af7d0 con 0x7fe32415a430 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 -- 172.21.15.136:0/216235927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32415a430 msgr2=0x7fe32415a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.301+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/216235927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32415a430 0x7fe32415a840 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe3240d00d0 tx=0x7fe31c01f1e0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- 172.21.15.136:0/216235927 shutdown_connections 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/216235927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32415a430 0x7fe32415a840 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/216235927 >> v2:172.21.15.157:3300/0 conn(0x7fe3240ca290 0x7fe3240ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- 172.21.15.136:0/216235927 >> 172.21.15.136:0/216235927 conn(0x7fe324070f70 msgr2=0x7fe324068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- 172.21.15.136:0/216235927 shutdown_connections 2024-04-25T11:33:45.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- 172.21.15.136:0/216235927 wait complete. 2024-04-25T11:33:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 Processor -- start 2024-04-25T11:33:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- start start 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3240ca290 0x7fe3241583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe32415a430 0x7fe32414b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe32415ad80 con 0x7fe3240b5060 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe324159f10 con 0x7fe32415a430 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.305+0000 7fe32bbe2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe32415d3f0 con 0x7fe3240ca290 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60352/0 (socket says 172.21.15.136:60352) 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 -- 172.21.15.136:0/3152736852 learned_addr learned my addr 172.21.15.136:0/3152736852 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32917d640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3240ca290 0x7fe3241583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32a17f640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe32415a430 0x7fe32414b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 -- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3240ca290 msgr2=0x7fe3241583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3240ca290 0x7fe3241583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 -- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe32415a430 msgr2=0x7fe32414b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe32415a430 0x7fe32414b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 -- 172.21.15.136:0/3152736852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe324158920 con 0x7fe3240b5060 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32997e640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c3a0 tx=0x7fe32000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3200052d0 con 0x7fe3240b5060 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe320005470 con 0x7fe3240b5060 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32415b0d0 con 0x7fe3240b5060 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320007260 con 0x7fe3240b5060 2024-04-25T11:33:45.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32415b930 con 0x7fe3240b5060 2024-04-25T11:33:45.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32406a350 con 0x7fe3240b5060 2024-04-25T11:33:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe320007400 con 0x7fe3240b5060 2024-04-25T11:33:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 0x7fe308071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fe32008ed40 con 0x7fe3240b5060 2024-04-25T11:33:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.309+0000 7fe32917d640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 0x7fe308071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:45.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.313+0000 7fe32917d640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 0x7fe308071940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe3240cb510 tx=0x7fe314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:45.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.317+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe32005b660 con 0x7fe3240b5060 2024-04-25T11:33:45.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3240cadd0 con 0x7fe3240b5060 2024-04-25T11:33:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe31a7fc640 1 -- 172.21.15.136:0/3152736852 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6099 (secure 0 0 0) 0x7fe32005af60 con 0x7fe3240b5060 2024-04-25T11:33:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:45.540 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:44.235005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5385,5356,5238]},"id":1}]} 2024-04-25T11:33:45.540 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 59 2024-04-25T11:33:45.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 msgr2=0x7fe308071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 0x7fe308071940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe3240cb510 tx=0x7fe314008040 comp rx=0 tx=0).stop 2024-04-25T11:33:45.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 msgr2=0x7fe3241f08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c3a0 tx=0x7fe32000c860 comp rx=0 tx=0).stop 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 shutdown_connections 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe32415a430 0x7fe32414b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe30806f430 0x7fe308071940 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe3240ca290 0x7fe3241583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 --2- 172.21.15.136:0/3152736852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3240b5060 0x7fe3241f08f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 >> 172.21.15.136:0/3152736852 conn(0x7fe324070f70 msgr2=0x7fe3240cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 shutdown_connections 2024-04-25T11:33:45.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:45.557+0000 7fe32bbe2640 1 -- 172.21.15.136:0/3152736852 wait complete. 2024-04-25T11:33:45.562 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:47.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:47.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 Processor -- start 2024-04-25T11:33:47.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 -- start start 2024-04-25T11:33:47.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5cc15d750 con 0x7fb5cc0b5060 2024-04-25T11:33:47.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5cc15da30 con 0x7fb5cc0ca290 2024-04-25T11:33:47.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.745+0000 7fb5d1798640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb5cc15dd10 con 0x7fb5cc15ab40 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60366/0 (socket says 172.21.15.136:60366) 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 -- 172.21.15.136:0/2137666424 learned_addr learned my addr 172.21.15.136:0/2137666424 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5ca7fc640 1 -- 172.21.15.136:0/2137666424 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596482979 0 0) 0x7fb5cc15d750 con 0x7fb5cc0b5060 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5ca7fc640 1 -- 172.21.15.136:0/2137666424 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5b8003530 con 0x7fb5cc0b5060 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 -- 172.21.15.136:0/2137666424 >> v1:172.21.15.157:6789/0 conn(0x7fb5cc0ca290 legacy=0x7fb5cc15a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 -- 172.21.15.136:0/2137666424 >> v1:172.21.15.136:6789/0 conn(0x7fb5cc0b5060 legacy=0x7fb5cc0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 -- 172.21.15.136:0/2137666424 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc15dff0 con 0x7fb5cc15ab40 2024-04-25T11:33:47.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5cb7fe640 1 --2- 172.21.15.136:0/2137666424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009e50 tx=0x7fb5c401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a300c6b683eb88a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5ca7fc640 1 -- 172.21.15.136:0/2137666424 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c40b4070 con 0x7fb5cc15ab40 2024-04-25T11:33:47.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5ca7fc640 1 -- 172.21.15.136:0/2137666424 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5c40af400 con 0x7fb5cc15ab40 2024-04-25T11:33:47.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5ca7fc640 1 -- 172.21.15.136:0/2137666424 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c40af7d0 con 0x7fb5cc15ab40 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 -- 172.21.15.136:0/2137666424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc15ab40 msgr2=0x7fb5cc0ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 --2- 172.21.15.136:0/2137666424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009e50 tx=0x7fb5c401f210 comp rx=0 tx=0).stop 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 -- 172.21.15.136:0/2137666424 shutdown_connections 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 --2- 172.21.15.136:0/2137666424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc0ca6a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 -- 172.21.15.136:0/2137666424 >> 172.21.15.136:0/2137666424 conn(0x7fb5cc070f70 msgr2=0x7fb5cc068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.749+0000 7fb5d1798640 1 -- 172.21.15.136:0/2137666424 shutdown_connections 2024-04-25T11:33:47.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.753+0000 7fb5d1798640 1 -- 172.21.15.136:0/2137666424 wait complete. 2024-04-25T11:33:47.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.753+0000 7fb5d1798640 1 Processor -- start 2024-04-25T11:33:47.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.753+0000 7fb5d1798640 1 -- start start 2024-04-25T11:33:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5cc0b5060 0x7fb5cc1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc15d3f0 con 0x7fb5cc0ca290 2024-04-25T11:33:47.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc159f10 con 0x7fb5cc15ab40 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb5cc165790 con 0x7fb5cc0b5060 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60376/0 (socket says 172.21.15.136:60376) 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 -- 172.21.15.136:0/3833706167 learned_addr learned my addr 172.21.15.136:0/3833706167 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5cb7fe640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5caffd640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5cc0b5060 0x7fb5cc1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 -- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5cc0b5060 msgr2=0x7fb5cc1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5cc0b5060 0x7fb5cc1483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 -- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5cc15ab40 msgr2=0x7fb5cc142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 -- 172.21.15.136:0/3833706167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc146b90 con 0x7fb5cc0ca290 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c3fff640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400c3f0 tx=0x7fb5b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4017070 con 0x7fb5cc0ca290 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b40052d0 con 0x7fb5cc0ca290 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 -- 172.21.15.136:0/3833706167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc146e70 con 0x7fb5cc0ca290 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4003840 con 0x7fb5cc0ca290 2024-04-25T11:33:47.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5d1798640 1 -- 172.21.15.136:0/3833706167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc147430 con 0x7fb5cc0ca290 2024-04-25T11:33:47.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a40052a0 con 0x7fb5cc0ca290 2024-04-25T11:33:47.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb5b401a070 con 0x7fb5cc0ca290 2024-04-25T11:33:47.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 0x7fb5ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:47.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.757+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fb5b408ebc0 con 0x7fb5cc0ca290 2024-04-25T11:33:47.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.761+0000 7fb5caffd640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 0x7fb5ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:47.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.761+0000 7fb5caffd640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 0x7fb5ac071940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0011f0 tx=0x7fb5bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:47.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.765+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb5b4005480 con 0x7fb5cc0ca290 2024-04-25T11:33:47.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5a4005860 con 0x7fb5cc0ca290 2024-04-25T11:33:47.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c37fe640 1 -- 172.21.15.136:0/3833706167 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6099 (secure 0 0 0) 0x7fb5b405b4e0 con 0x7fb5cc0ca290 2024-04-25T11:33:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:47.968 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:44.235005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5385,5356,5238]},"id":1}]} 2024-04-25T11:33:47.968 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 59 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 msgr2=0x7fb5ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 0x7fb5ac071940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0011f0 tx=0x7fb5bc009040 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 msgr2=0x7fb5cc141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400c3f0 tx=0x7fb5b400c8b0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 shutdown_connections 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb5cc15ab40 0x7fb5cc142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5cc0ca290 0x7fb5cc141c30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb5ac06f430 0x7fb5ac071940 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 --2- 172.21.15.136:0/3833706167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb5cc0b5060 0x7fb5cc1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 >> 172.21.15.136:0/3833706167 conn(0x7fb5cc070f70 msgr2=0x7fb5cc148db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 shutdown_connections 2024-04-25T11:33:47.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:47.985+0000 7fb5c17fa640 1 -- 172.21.15.136:0/3833706167 wait complete. 2024-04-25T11:33:47.993 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:33:49.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:50.158 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:33:50.172+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:33:50.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 Processor -- start 2024-04-25T11:33:50.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 -- start start 2024-04-25T11:33:50.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f895415c0d0 0x7f89540cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8954160370 0x7f8954166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f895415c6f0 con 0x7f895415ff60 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f89540c6310 con 0x7f895415c0d0 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8959a29640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f895415c990 con 0x7f8954160370 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f895415c0d0 0x7f89540cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f895415c0d0 0x7f89540cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60386/0 (socket says 172.21.15.136:60386) 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f894bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f895415ff60 0x7f89540ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46316/0 (socket says 172.21.15.136:46316) 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 -- 172.21.15.136:0/3339616581 learned_addr learned my addr 172.21.15.136:0/3339616581 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f89537fe640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8954160370 0x7f8954166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 -- 172.21.15.136:0/3339616581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8954160370 msgr2=0x7f8954166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8954160370 0x7f8954166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 -- 172.21.15.136:0/3339616581 >> v1:172.21.15.136:6789/0 conn(0x7f895415ff60 legacy=0x7f89540ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f8952ffd640 1 -- 172.21.15.136:0/3339616581 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895415cc30 con 0x7f895415c0d0 2024-04-25T11:33:50.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.181+0000 7f89527fc640 1 -- 172.21.15.136:0/3339616581 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130043216 0 0) 0x7f895415c6f0 con 0x7f895415ff60 2024-04-25T11:33:50.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8952ffd640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415c0d0 0x7f89540cb720 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8944009870 tx=0x7f894401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d70f75e513e4343 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:50.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f89527fc640 1 -- 172.21.15.136:0/3339616581 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440b5070 con 0x7f895415c0d0 2024-04-25T11:33:50.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f89527fc640 1 -- 172.21.15.136:0/3339616581 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89440a8a50 con 0x7f895415c0d0 2024-04-25T11:33:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f89527fc640 1 -- 172.21.15.136:0/3339616581 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440a8e20 con 0x7f895415c0d0 2024-04-25T11:33:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 -- 172.21.15.136:0/3339616581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415c0d0 msgr2=0x7f89540cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415c0d0 0x7f89540cb720 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8944009870 tx=0x7f894401ed50 comp rx=0 tx=0).stop 2024-04-25T11:33:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 -- 172.21.15.136:0/3339616581 shutdown_connections 2024-04-25T11:33:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8954160370 0x7f8954166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 --2- 172.21.15.136:0/3339616581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415c0d0 0x7f89540cb720 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 -- 172.21.15.136:0/3339616581 >> 172.21.15.136:0/3339616581 conn(0x7f895406f9d0 msgr2=0x7f895406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 -- 172.21.15.136:0/3339616581 shutdown_connections 2024-04-25T11:33:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.185+0000 7f8959a29640 1 -- 172.21.15.136:0/3339616581 wait complete. 2024-04-25T11:33:50.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 Processor -- start 2024-04-25T11:33:50.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- start start 2024-04-25T11:33:50.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f895415c0d0 0x7f89541f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8954160370 0x7f895414ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f89540cbc60 con 0x7f895415ff60 2024-04-25T11:33:50.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f895415fa40 con 0x7f8954160370 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f895415cf10 con 0x7f895415c0d0 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:60388/0 (socket says 172.21.15.136:60388) 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 -- 172.21.15.136:0/1298572902 learned_addr learned my addr 172.21.15.136:0/1298572902 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8952ffd640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f895415c0d0 0x7f89541f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f89537fe640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8954160370 0x7f895414ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 -- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f895415c0d0 msgr2=0x7f89541f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f895415c0d0 0x7f89541f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 -- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8954160370 msgr2=0x7f895414ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8954160370 0x7f895414ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 -- 172.21.15.136:0/1298572902 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89541f2c90 con 0x7f895415ff60 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f894bfff640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f893c00c4b0 tx=0x7f893c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c018070 con 0x7f895415ff60 2024-04-25T11:33:50.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893c007500 con 0x7f895415ff60 2024-04-25T11:33:50.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89541f5700 con 0x7f895415ff60 2024-04-25T11:33:50.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c004a30 con 0x7f895415ff60 2024-04-25T11:33:50.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.189+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541f5f80 con 0x7f895415ff60 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895406a350 con 0x7f895415ff60 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f893c01b070 con 0x7f895415ff60 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8950ff9640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 0x7f8934071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f893c08eb60 con 0x7f895415ff60 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8952ffd640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 0x7f8934071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:50.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.193+0000 7f8952ffd640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 0x7f8934071910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f89440b2000 tx=0x7f89440aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:50.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.197+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f893c05ada0 con 0x7f895415ff60 2024-04-25T11:33:50.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.425+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89540c6c10 con 0x7f895415ff60 2024-04-25T11:33:50.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.425+0000 7f8950ff9640 1 -- 172.21.15.136:0/1298572902 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6104 (secure 0 0 0) 0x7f893c05f0a0 con 0x7f895415ff60 2024-04-25T11:33:50.410 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:50.410 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:33:50.411 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 60 2024-04-25T11:33:50.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 msgr2=0x7f8934071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 0x7f8934071910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f89440b2000 tx=0x7f89440aa070 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 msgr2=0x7f89541e7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f893c00c4b0 tx=0x7f893c00c970 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 shutdown_connections 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8954160370 0x7f895414ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f895415ff60 0x7f89541e7980 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f893406f400 0x7f8934071910 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 --2- 172.21.15.136:0/1298572902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f895415c0d0 0x7f89541f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 >> 172.21.15.136:0/1298572902 conn(0x7f895406f9d0 msgr2=0x7f8954159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 shutdown_connections 2024-04-25T11:33:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:50.429+0000 7f8959a29640 1 -- 172.21.15.136:0/1298572902 wait complete. 2024-04-25T11:33:50.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=5036 2024-04-25T11:33:50.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2024-04-25T11:33:53.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:33:53.055 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:33:57.434 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-25T11:33:57.435 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T11:33:57.435 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T11:33:57.438 INFO:tasks.ceph.mds.b:Started 2024-04-25T11:33:57.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:33:57.545 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:33:57.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.629+0000 7f5fc6662640 1 Processor -- start 2024-04-25T11:33:57.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.629+0000 7f5fc6662640 1 -- start start 2024-04-25T11:33:57.616 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc6662640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5fc0154060 0x7f5fc00c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc6662640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5fc015aa00 0x7f5fc0164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc6662640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fc00c8d40 con 0x7f5fc015a5f0 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc6662640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5fc00c8fd0 con 0x7f5fc0154060 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc6662640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5fc00c3880 con 0x7f5fc015aa00 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5fc0154060 0x7f5fc00c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5fc0154060 0x7f5fc00c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37912/0 (socket says 172.21.15.136:37912) 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5fc015a5f0 0x7f5fc00c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:51362/0 (socket says 172.21.15.136:51362) 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 -- 172.21.15.136:0/1842195411 learned_addr learned my addr 172.21.15.136:0/1842195411 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fc4bff640 1 --2- 172.21.15.136:0/1842195411 >> v2:172.21.15.157:3300/0 conn(0x7f5fc015aa00 0x7f5fc0164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 -- 172.21.15.136:0/1842195411 >> v2:172.21.15.157:3300/0 conn(0x7f5fc015aa00 msgr2=0x7f5fc0164700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:33:57.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 --2- 172.21.15.136:0/1842195411 >> v2:172.21.15.157:3300/0 conn(0x7f5fc015aa00 0x7f5fc0164700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 -- 172.21.15.136:0/1842195411 >> v1:172.21.15.136:6789/0 conn(0x7f5fc015a5f0 legacy=0x7f5fc00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbeffd640 1 -- 172.21.15.136:0/1842195411 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692157234 0 0) 0x7f5fc00c8d40 con 0x7f5fc015a5f0 2024-04-25T11:33:57.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.633+0000 7f5fbffff640 1 -- 172.21.15.136:0/1842195411 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc006d9b0 con 0x7f5fc0154060 2024-04-25T11:33:57.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fbffff640 1 --2- 172.21.15.136:0/1842195411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc0154060 0x7f5fc00c8800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c80e0 tx=0x7f5fb001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4939cd1bbed911f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:57.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fbeffd640 1 -- 172.21.15.136:0/1842195411 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00b5070 con 0x7f5fc0154060 2024-04-25T11:33:57.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fbeffd640 1 -- 172.21.15.136:0/1842195411 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb00a8a10 con 0x7f5fc0154060 2024-04-25T11:33:57.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fbeffd640 1 -- 172.21.15.136:0/1842195411 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00a8de0 con 0x7f5fc0154060 2024-04-25T11:33:57.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 -- 172.21.15.136:0/1842195411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc0154060 msgr2=0x7f5fc00c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 --2- 172.21.15.136:0/1842195411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc0154060 0x7f5fc00c8800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00c80e0 tx=0x7f5fb001ed10 comp rx=0 tx=0).stop 2024-04-25T11:33:57.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 -- 172.21.15.136:0/1842195411 shutdown_connections 2024-04-25T11:33:57.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 --2- 172.21.15.136:0/1842195411 >> v2:172.21.15.157:3300/0 conn(0x7f5fc015aa00 0x7f5fc0164700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 --2- 172.21.15.136:0/1842195411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc0154060 0x7f5fc00c8800 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 -- 172.21.15.136:0/1842195411 >> 172.21.15.136:0/1842195411 conn(0x7f5fc00ca5a0 msgr2=0x7f5fc00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:57.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 -- 172.21.15.136:0/1842195411 shutdown_connections 2024-04-25T11:33:57.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.637+0000 7f5fc6662640 1 -- 172.21.15.136:0/1842195411 wait complete. 2024-04-25T11:33:57.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 Processor -- start 2024-04-25T11:33:57.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- start start 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 0x7f5fc014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fc015aa00 0x7f5fc014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc006df30 con 0x7f5fc015a5f0 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc0154470 con 0x7f5fc0154060 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5fc015a0d0 con 0x7f5fc015aa00 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37922/0 (socket says 172.21.15.136:37922) 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 -- 172.21.15.136:0/3213477673 learned_addr learned my addr 172.21.15.136:0/3213477673 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:33:57.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbffff640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 0x7f5fc014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc4bff640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fc015aa00 0x7f5fc014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 -- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fc015aa00 msgr2=0x7f5fc014c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fc015aa00 0x7f5fc014c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 -- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 msgr2=0x7f5fc014fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 0x7f5fc014fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 -- 172.21.15.136:0/3213477673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc01f6470 con 0x7f5fc015a5f0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbffff640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 0x7f5fc014fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbf7fe640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c410 tx=0x7f5fb400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40074e0 con 0x7f5fc015a5f0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb40047f0 con 0x7f5fc015a5f0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc0075000 con 0x7f5fc015a5f0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4007680 con 0x7f5fc015a5f0 2024-04-25T11:33:57.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc014e810 con 0x7f5fc015a5f0 2024-04-25T11:33:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.641+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc006a350 con 0x7f5fc015a5f0 2024-04-25T11:33:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.645+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5fb4011070 con 0x7f5fc015a5f0 2024-04-25T11:33:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.645+0000 7f5fbcff9640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 0x7f5fac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:33:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.645+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f5fb408f420 con 0x7f5fc015a5f0 2024-04-25T11:33:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.645+0000 7f5fbffff640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 0x7f5fac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:33:57.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.645+0000 7f5fbffff640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 0x7f5fac071940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00b2000 tx=0x7f5fb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:33:57.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.649+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5fb405bdd0 con 0x7f5fc015a5f0 2024-04-25T11:33:57.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.873+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc00c85e0 con 0x7f5fc015a5f0 2024-04-25T11:33:57.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.873+0000 7f5fbcff9640 1 -- 172.21.15.136:0/3213477673 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6104 (secure 0 0 0) 0x7f5fb405b6d0 con 0x7f5fc015a5f0 2024-04-25T11:33:57.858 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:33:57.859 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5238,5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:33:57.859 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 60 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 msgr2=0x7f5fac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 0x7f5fac071940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00b2000 tx=0x7f5fb0008040 comp rx=0 tx=0).stop 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 msgr2=0x7f5fc01f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400c410 tx=0x7f5fb400c8d0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 shutdown_connections 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5fac06f430 0x7f5fac071940 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fc015aa00 0x7f5fc014c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc015a5f0 0x7f5fc01f5f30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 --2- 172.21.15.136:0/3213477673 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fc0154060 0x7f5fc014fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:33:57.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 >> 172.21.15.136:0/3213477673 conn(0x7f5fc00ca5a0 msgr2=0x7f5fc00c2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:33:57.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 shutdown_connections 2024-04-25T11:33:57.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:33:57.877+0000 7f5fc6662640 1 -- 172.21.15.136:0/3213477673 wait complete. 2024-04-25T11:33:57.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T11:33:59.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:34:00.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 Processor -- start 2024-04-25T11:34:00.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 -- start start 2024-04-25T11:34:00.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f67880c9b50 0x7f678815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f678815c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67880c8a20 con 0x7f67880ca010 2024-04-25T11:34:00.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f678806c670 con 0x7f67880c9b50 2024-04-25T11:34:00.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f679025a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67880c8d00 con 0x7f678806d210 2024-04-25T11:34:00.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678dff6640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f67880ca010 0x7f678806cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:51370/0 (socket says 172.21.15.136:51370) 2024-04-25T11:34:00.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678dff6640 1 -- 172.21.15.136:0/1102282224 learned_addr learned my addr 172.21.15.136:0/1102282224 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678d7f5640 1 --2- 172.21.15.136:0/1102282224 >> v2:172.21.15.157:3300/0 conn(0x7f67880c9b50 0x7f678815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884837895 0 0) 0x7f67880c8a20 con 0x7f67880ca010 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678e7f7640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f678815c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6774003530 con 0x7f67880ca010 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 439610590 0 0) 0x7f6774003530 con 0x7f67880ca010 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 msgr2=0x7f678815c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f678815c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 >> v2:172.21.15.157:3300/0 conn(0x7f67880c9b50 msgr2=0x7f678815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 --2- 172.21.15.136:0/1102282224 >> v2:172.21.15.157:3300/0 conn(0x7f67880c9b50 0x7f678815bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6788074d10 con 0x7f67880ca010 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f67840016a0 con 0x7f67880ca010 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 >> v1:172.21.15.136:6789/0 conn(0x7f67880ca010 legacy=0x7f678806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f67740039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6774003f40 0x7f6774004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6774004d50 0x7f6774005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f678806c670 con 0x7f6774003f40 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67880c8d00 con 0x7f6774004d50 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.048+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f677400ba20 con 0x7f678806d210 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6774003f40 0x7f6774004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678dff6640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f67740039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678e7f7640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6774004d50 0x7f6774005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 -- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 msgr2=0x7f67740039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f678806d210 0x7f67740039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 -- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6774004d50 msgr2=0x7f6774005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6774004d50 0x7f6774005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 -- 172.21.15.136:0/1102282224 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6774003530 con 0x7f6774003f40 2024-04-25T11:34:00.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678d7f5640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6774003f40 0x7f6774004810 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6788072cb0 tx=0x7f677801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabf2d530f7803dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:00.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780b2070 con 0x7f6774003f40 2024-04-25T11:34:00.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67780a5620 con 0x7f6774003f40 2024-04-25T11:34:00.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.052+0000 7f678cff4640 1 -- 172.21.15.136:0/1102282224 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780096b0 con 0x7f6774003f40 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- 172.21.15.136:0/1102282224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6774003f40 msgr2=0x7f6774004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 --2- 172.21.15.136:0/1102282224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6774003f40 0x7f6774004810 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6788072cb0 tx=0x7f677801ca30 comp rx=0 tx=0).stop 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f678e7f7640 1 -- 172.21.15.136:0/1102282224 reap_dead start 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- 172.21.15.136:0/1102282224 shutdown_connections 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- 172.21.15.136:0/1102282224 >> 172.21.15.136:0/1102282224 conn(0x7f67880ca5a0 msgr2=0x7f67880ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- 172.21.15.136:0/1102282224 shutdown_connections 2024-04-25T11:34:00.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- 172.21.15.136:0/1102282224 wait complete. 2024-04-25T11:34:00.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 Processor -- start 2024-04-25T11:34:00.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.056+0000 7f679025a640 1 -- start start 2024-04-25T11:34:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6774003f40 0x7f67881f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67881e7f20 0x7f67881e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f678815c520 con 0x7f678814c130 2024-04-25T11:34:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67880c9240 con 0x7f67881e7f20 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f678815c6a0 con 0x7f6774003f40 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:37940/0 (socket says 172.21.15.136:37940) 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 -- 172.21.15.136:0/938242648 learned_addr learned my addr 172.21.15.136:0/938242648 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678dff6640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6774003f40 0x7f67881f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678d7f5640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67881e7f20 0x7f67881e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 -- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6774003f40 msgr2=0x7f67881f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6774003f40 0x7f67881f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 -- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67881e7f20 msgr2=0x7f67881e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67881e7f20 0x7f67881e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 -- 172.21.15.136:0/938242648 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67881e6080 con 0x7f678814c130 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f678e7f7640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f678806e920 tx=0x7f67800115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780011db0 con 0x7f678814c130 2024-04-25T11:34:00.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678001b070 con 0x7f678814c130 2024-04-25T11:34:00.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67881e7100 con 0x7f678814c130 2024-04-25T11:34:00.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67800039a0 con 0x7f678814c130 2024-04-25T11:34:00.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67881e76c0 con 0x7f678814c130 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.060+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f678806a350 con 0x7f678814c130 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.064+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6780003b40 con 0x7f678814c130 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.064+0000 7f677e7fc640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 0x7f676c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.064+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f6780090030 con 0x7f678814c130 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.064+0000 7f678dff6640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 0x7f676c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:34:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.064+0000 7f678dff6640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 0x7f676c071940 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c10 tx=0x7f6784009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:34:00.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.072+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6780025020 con 0x7f678814c130 2024-04-25T11:34:00.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.292+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67881e6e60 con 0x7f678814c130 2024-04-25T11:34:00.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.292+0000 7f677e7fc640 1 -- 172.21.15.136:0/938242648 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7f678005c9e0 con 0x7f678814c130 2024-04-25T11:34:00.278 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:34:00.279 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:34:00.279 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 msgr2=0x7f676c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 0x7f676c071940 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6784000c10 tx=0x7f6784009040 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 msgr2=0x7f678814c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f678806e920 tx=0x7f67800115a0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 shutdown_connections 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f678814c130 0x7f678814c540 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67881e7f20 0x7f67881e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f676c06f430 0x7f676c071940 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 --2- 172.21.15.136:0/938242648 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6774003f40 0x7f67881f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 >> 172.21.15.136:0/938242648 conn(0x7f67880ca5a0 msgr2=0x7f6788075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 shutdown_connections 2024-04-25T11:34:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:34:00.296+0000 7f679025a640 1 -- 172.21.15.136:0/938242648 wait complete. 2024-04-25T11:34:00.300 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-25T11:34:00.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2024-04-25T11:34:23.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:34:23.329 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:34:53.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:34:53.375 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:35:23.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:35:23.488 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:35:42.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:42.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 Processor -- start 2024-04-25T11:35:42.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 -- start start 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fde4c154060 0x7fde4c0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde4c15aa00 0x7fde4c0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fde4c0c8fd0 con 0x7fde4c15aa00 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fde4c0c8d40 con 0x7fde4c154060 2024-04-25T11:35:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde53804640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde4c0c3880 con 0x7fde4c15a5f0 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde50d9f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde515a0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fde4c154060 0x7fde4c0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde50d9f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58624/0 (socket says 172.21.15.136:58624) 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde50d9f640 1 -- 172.21.15.136:0/2778202037 learned_addr learned my addr 172.21.15.136:0/2778202037 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 --2- 172.21.15.136:0/2778202037 >> v2:172.21.15.136:3300/0 conn(0x7fde4c15aa00 0x7fde4c0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 -- 172.21.15.136:0/2778202037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 msgr2=0x7fde4c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 -- 172.21.15.136:0/2778202037 >> v2:172.21.15.157:3300/0 conn(0x7fde4c154060 msgr2=0x7fde4c0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:42.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 --2- 172.21.15.136:0/2778202037 >> v2:172.21.15.157:3300/0 conn(0x7fde4c154060 0x7fde4c0c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde51da1640 1 -- 172.21.15.136:0/2778202037 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4c06d9b0 con 0x7fde4c15aa00 2024-04-25T11:35:42.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.494+0000 7fde50d9f640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:42.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde51da1640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15aa00 0x7fde4c0c76f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde4c06db80 tx=0x7fde4401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cebb6d9fab4c855 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde43fff640 1 -- 172.21.15.136:0/2778202037 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440b5070 con 0x7fde4c15aa00 2024-04-25T11:35:42.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde43fff640 1 -- 172.21.15.136:0/2778202037 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde440b0440 con 0x7fde4c15aa00 2024-04-25T11:35:42.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde43fff640 1 -- 172.21.15.136:0/2778202037 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440b0810 con 0x7fde4c15aa00 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 -- 172.21.15.136:0/2778202037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15aa00 msgr2=0x7fde4c0c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15aa00 0x7fde4c0c76f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fde4c06db80 tx=0x7fde4401f110 comp rx=0 tx=0).stop 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 -- 172.21.15.136:0/2778202037 shutdown_connections 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15aa00 0x7fde4c0c76f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 --2- 172.21.15.136:0/2778202037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15a5f0 0x7fde4c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 --2- 172.21.15.136:0/2778202037 >> v2:172.21.15.157:3300/0 conn(0x7fde4c154060 0x7fde4c0c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 -- 172.21.15.136:0/2778202037 >> 172.21.15.136:0/2778202037 conn(0x7fde4c0ca5a0 msgr2=0x7fde4c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 -- 172.21.15.136:0/2778202037 shutdown_connections 2024-04-25T11:35:42.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.498+0000 7fde53804640 1 -- 172.21.15.136:0/2778202037 wait complete. 2024-04-25T11:35:42.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 Processor -- start 2024-04-25T11:35:42.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 -- start start 2024-04-25T11:35:42.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde4c154060 0x7fde4c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15aa00 0x7fde4c15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c06df30 con 0x7fde4c15a5f0 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde4c154470 con 0x7fde4c154060 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde53804640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde4c15a0d0 con 0x7fde4c15aa00 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde50d9f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde50d9f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:57564/0 (socket says 172.21.15.136:57564) 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde50d9f640 1 -- 172.21.15.136:0/502173118 learned_addr learned my addr 172.21.15.136:0/502173118 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde515a0640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde4c154060 0x7fde4c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.502+0000 7fde51da1640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15aa00 0x7fde4c15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 -- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15aa00 msgr2=0x7fde4c15d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15aa00 0x7fde4c15d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 -- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde4c154060 msgr2=0x7fde4c0c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde4c154060 0x7fde4c0c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 -- 172.21.15.136:0/502173118 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4c1f1e60 con 0x7fde4c15a5f0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde50d9f640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e900 tx=0x7fde3c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00ca00 con 0x7fde4c15a5f0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c00cba0 con 0x7fde4c15a5f0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c004030 con 0x7fde4c15a5f0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4c1ee0f0 con 0x7fde4c15a5f0 2024-04-25T11:35:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde4c1ee6b0 con 0x7fde4c15a5f0 2024-04-25T11:35:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4c06a350 con 0x7fde4c15a5f0 2024-04-25T11:35:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde3c0041d0 con 0x7fde4c15a5f0 2024-04-25T11:35:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 0x7fde30071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fde3c08fc70 con 0x7fde4c15a5f0 2024-04-25T11:35:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.506+0000 7fde515a0640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 0x7fde30071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.510+0000 7fde515a0640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 0x7fde30071910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fde480099d0 tx=0x7fde48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.514+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde3c017020 con 0x7fde4c15a5f0 2024-04-25T11:35:42.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.734+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde4c1f2170 con 0x7fde4c15a5f0 2024-04-25T11:35:42.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.734+0000 7fde41ffb640 1 -- 172.21.15.136:0/502173118 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7fde3c05c500 con 0x7fde4c15a5f0 2024-04-25T11:35:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:42.722 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:42.722 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 msgr2=0x7fde30071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 0x7fde30071910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fde480099d0 tx=0x7fde48008040 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 msgr2=0x7fde4c14c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e900 tx=0x7fde3c00edc0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 shutdown_connections 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde3006f400 0x7fde30071910 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde4c15aa00 0x7fde4c15d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde4c15a5f0 0x7fde4c14c980 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 --2- 172.21.15.136:0/502173118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde4c154060 0x7fde4c0c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 >> 172.21.15.136:0/502173118 conn(0x7fde4c0ca5a0 msgr2=0x7fde4c14d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 shutdown_connections 2024-04-25T11:35:42.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.738+0000 7fde53804640 1 -- 172.21.15.136:0/502173118 wait complete. 2024-04-25T11:35:42.744 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=4) 2024-04-25T11:35:42.745 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T11:35:42.745 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:35:42.757 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T11:35:42.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:42.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.962+0000 7fe764758640 1 Processor -- start 2024-04-25T11:35:42.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.962+0000 7fe764758640 1 -- start start 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe75c0c8cd0 0x7fe75c14f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe75c0ce260 0x7fe75c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe75c0c8550 con 0x7fe75c0ce260 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe75c0cbad0 con 0x7fe75c0c8cd0 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe764758640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c0c7c20 con 0x7fe75c0ca010 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe75c0ce260 0x7fe75c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe75c0ce260 0x7fe75c0ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:57570/0 (socket says 172.21.15.136:57570) 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 -- 172.21.15.136:0/1701123338 learned_addr learned my addr 172.21.15.136:0/1701123338 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe762cf5640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe7624f4640 1 --2- 172.21.15.136:0/1701123338 >> v2:172.21.15.157:3300/0 conn(0x7fe75c0c8cd0 0x7fe75c14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 -- 172.21.15.136:0/1701123338 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 msgr2=0x7fe75c0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c0ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 -- 172.21.15.136:0/1701123338 >> v2:172.21.15.157:3300/0 conn(0x7fe75c0c8cd0 msgr2=0x7fe75c14f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 --2- 172.21.15.136:0/1701123338 >> v2:172.21.15.157:3300/0 conn(0x7fe75c0c8cd0 0x7fe75c14f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.966+0000 7fe761cf3640 1 -- 172.21.15.136:0/1701123338 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c0c7f00 con 0x7fe75c0ce260 2024-04-25T11:35:42.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe761cf3640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c0ce670 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0734b0 tx=0x7fe74c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af36966614c06c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe7614f2640 1 -- 172.21.15.136:0/1701123338 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0b5070 con 0x7fe75c0ce260 2024-04-25T11:35:42.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe7614f2640 1 -- 172.21.15.136:0/1701123338 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74c0a89d0 con 0x7fe75c0ce260 2024-04-25T11:35:42.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe7614f2640 1 -- 172.21.15.136:0/1701123338 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0a8da0 con 0x7fe75c0ce260 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- 172.21.15.136:0/1701123338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 msgr2=0x7fe75c0ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c0ce670 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0734b0 tx=0x7fe74c01ed10 comp rx=0 tx=0).stop 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- 172.21.15.136:0/1701123338 shutdown_connections 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c0ce670 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 --2- 172.21.15.136:0/1701123338 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c0ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 --2- 172.21.15.136:0/1701123338 >> v2:172.21.15.157:3300/0 conn(0x7fe75c0c8cd0 0x7fe75c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- 172.21.15.136:0/1701123338 >> 172.21.15.136:0/1701123338 conn(0x7fe75c06f9d0 msgr2=0x7fe75c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- 172.21.15.136:0/1701123338 shutdown_connections 2024-04-25T11:35:42.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- 172.21.15.136:0/1701123338 wait complete. 2024-04-25T11:35:42.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 Processor -- start 2024-04-25T11:35:42.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.970+0000 7fe764758640 1 -- start start 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe75c0c8cd0 0x7fe75c1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c15baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0cb720 con 0x7fe75c0ce260 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe75c0ca960 con 0x7fe75c0c8cd0 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe75c1f32f0 con 0x7fe75c0ca010 2024-04-25T11:35:42.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:57586/0 (socket says 172.21.15.136:57586) 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 -- 172.21.15.136:0/2844354770 learned_addr learned my addr 172.21.15.136:0/2844354770 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe7624f4640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe75c0c8cd0 0x7fe75c1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe761cf3640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c15baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 -- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 msgr2=0x7fe75c15baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c15baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 -- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe75c0c8cd0 msgr2=0x7fe75c1f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe75c0c8cd0 0x7fe75c1f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 -- 172.21.15.136:0/2844354770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75c1f2e20 con 0x7fe75c0ce260 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe762cf5640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe754007e90 tx=0x7fe75400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754014420 con 0x7fe75c0ce260 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe754019070 con 0x7fe75c0ce260 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754014800 con 0x7fe75c0ce260 2024-04-25T11:35:42.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75c14d050 con 0x7fe75c0ce260 2024-04-25T11:35:42.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75c14d610 con 0x7fe75c0ce260 2024-04-25T11:35:42.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.974+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75c06a350 con 0x7fe75c0ce260 2024-04-25T11:35:42.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.978+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe7540149a0 con 0x7fe75c0ce260 2024-04-25T11:35:42.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.978+0000 7fe752ffd640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 0x7fe740071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:42.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.978+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fe75408ff40 con 0x7fe75c0ce260 2024-04-25T11:35:42.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.978+0000 7fe7624f4640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 0x7fe740071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:42.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.978+0000 7fe7624f4640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 0x7fe740071910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe7580045f0 tx=0x7fe758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:42.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:42.982+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe75405c2a0 con 0x7fe75c0ce260 2024-04-25T11:35:43.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.202+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe75c14ee00 con 0x7fe75c0ce260 2024-04-25T11:35:43.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.202+0000 7fe752ffd640 1 -- 172.21.15.136:0/2844354770 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7fe7540605a0 con 0x7fe75c0ce260 2024-04-25T11:35:43.189 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:43.189 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:43.189 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:43.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 msgr2=0x7fe740071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 0x7fe740071910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe7580045f0 tx=0x7fe758008040 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 msgr2=0x7fe75c148bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe754007e90 tx=0x7fe75400e4b0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 shutdown_connections 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe75c0ce260 0x7fe75c148bb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe74006f400 0x7fe740071910 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe75c0ca010 0x7fe75c15baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 --2- 172.21.15.136:0/2844354770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe75c0c8cd0 0x7fe75c1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 >> 172.21.15.136:0/2844354770 conn(0x7fe75c06f9d0 msgr2=0x7fe75c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 shutdown_connections 2024-04-25T11:35:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:43.206+0000 7fe764758640 1 -- 172.21.15.136:0/2844354770 wait complete. 2024-04-25T11:35:43.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:45.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:45.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.386+0000 7f5ac8e70640 1 Processor -- start 2024-04-25T11:35:45.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.386+0000 7f5ac8e70640 1 -- start start 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac8e70640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5ac40c8cd0 0x7f5ac414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac8e70640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac8e70640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac40cbad0 con 0x7f5ac40ca010 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac8e70640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac40c8550 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac8e70640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac40c7c20 con 0x7f5ac40ce260 2024-04-25T11:35:45.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac2d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5ac40ca010 0x7f5ac40ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44828/0 (socket says 172.21.15.136:44828) 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac2d9d640 1 -- 172.21.15.136:0/1137978632 learned_addr learned my addr 172.21.15.136:0/1137978632 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 --2- 172.21.15.136:0/1137978632 >> v2:172.21.15.136:3300/0 conn(0x7f5ac40c8cd0 0x7f5ac414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac1d9b640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac159a640 1 -- 172.21.15.136:0/1137978632 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518652934 0 0) 0x7f5ac40cbad0 con 0x7f5ac40ca010 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac159a640 1 -- 172.21.15.136:0/1137978632 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aac003530 con 0x7f5ac40ca010 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 -- 172.21.15.136:0/1137978632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 msgr2=0x7f5ac40ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac40ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 -- 172.21.15.136:0/1137978632 >> v1:172.21.15.136:6789/0 conn(0x7f5ac40ca010 legacy=0x7f5ac40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 -- 172.21.15.136:0/1137978632 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac40c7f00 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac259c640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac414f680 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40c86d0 tx=0x7f5ab401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4327676f1352fe91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac159a640 1 -- 172.21.15.136:0/1137978632 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b5070 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac159a640 1 -- 172.21.15.136:0/1137978632 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40a8990 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.390+0000 7f5ac159a640 1 -- 172.21.15.136:0/1137978632 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40a8d60 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- 172.21.15.136:0/1137978632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40c8cd0 msgr2=0x7f5ac414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac414f680 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40c86d0 tx=0x7f5ab401ec90 comp rx=0 tx=0).stop 2024-04-25T11:35:45.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- 172.21.15.136:0/1137978632 shutdown_connections 2024-04-25T11:35:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac40ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/1137978632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac414f680 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- 172.21.15.136:0/1137978632 >> 172.21.15.136:0/1137978632 conn(0x7f5ac406f9d0 msgr2=0x7f5ac406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- 172.21.15.136:0/1137978632 shutdown_connections 2024-04-25T11:35:45.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- 172.21.15.136:0/1137978632 wait complete. 2024-04-25T11:35:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 Processor -- start 2024-04-25T11:35:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- start start 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac41f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac4154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac40cb720 con 0x7f5ac40ca010 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac414fbc0 con 0x7f5ac40c8cd0 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.394+0000 7f5ac8e70640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac41f0c20 con 0x7f5ac40ce260 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34782/0 (socket says 172.21.15.136:34782) 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 -- 172.21.15.136:0/2898654579 learned_addr learned my addr 172.21.15.136:0/2898654579 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac259c640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac41f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac2d9d640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac4154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 -- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 msgr2=0x7f5ac4154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac4154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 -- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac40c8cd0 msgr2=0x7f5ac41f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac41f04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 -- 172.21.15.136:0/2898654579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac41f0a10 con 0x7f5ac40ca010 2024-04-25T11:35:45.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac2d9d640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac4154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:35:45.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac1d9b640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800d620 tx=0x7f5ab800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800b620 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab8019070 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac40c99c0 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab800b9e0 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac41f4580 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac406a350 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ab800bb80 con 0x7f5ac40ca010 2024-04-25T11:35:45.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 0x7f5aa4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:45.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f5ab808fd30 con 0x7f5ac40ca010 2024-04-25T11:35:45.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.398+0000 7f5ac259c640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 0x7f5aa4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:45.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.402+0000 7f5ac259c640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 0x7f5aa4071910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40b2000 tx=0x7f5ab4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:45.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.406+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ab805c650 con 0x7f5ac40ca010 2024-04-25T11:35:45.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.634+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac4148ac0 con 0x7f5ac40ca010 2024-04-25T11:35:45.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.634+0000 7f5ab2ffd640 1 -- 172.21.15.136:0/2898654579 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7f5ab805bf50 con 0x7f5ac40ca010 2024-04-25T11:35:45.622 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:45.622 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:45.622 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:45.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 msgr2=0x7f5aa4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 0x7f5aa4071910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40b2000 tx=0x7f5ab4008040 comp rx=0 tx=0).stop 2024-04-25T11:35:45.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 msgr2=0x7f5ac41547b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:45.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800d620 tx=0x7f5ab800dae0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 shutdown_connections 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5aa406f400 0x7f5aa4071910 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5ac40ce260 0x7f5ac4154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ac40ca010 0x7f5ac41547b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 --2- 172.21.15.136:0/2898654579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ac40c8cd0 0x7f5ac41f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 >> 172.21.15.136:0/2898654579 conn(0x7f5ac406f9d0 msgr2=0x7f5ac4071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 shutdown_connections 2024-04-25T11:35:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:45.638+0000 7f5ac8e70640 1 -- 172.21.15.136:0/2898654579 wait complete. 2024-04-25T11:35:45.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:47.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:47.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.802+0000 7f4daccc4640 1 Processor -- start 2024-04-25T11:35:47.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.802+0000 7f4daccc4640 1 -- start start 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4daccc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da8156750 0x7f4da80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4daccc4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca7d0 0x7f4da80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4daccc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da815c410 con 0x7f4da80cf9d0 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4daccc4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f4da80cfde0 con 0x7f4da80ca7d0 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4daccc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4da815c6b0 con 0x7f4da8156750 2024-04-25T11:35:47.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca7d0 0x7f4da80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca7d0 0x7f4da80cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34784/0 (socket says 172.21.15.136:34784) 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 -- 172.21.15.136:0/4246516579 learned_addr learned my addr 172.21.15.136:0/4246516579 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da659c640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da8156750 0x7f4da80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 -- 172.21.15.136:0/4246516579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da8156750 msgr2=0x7f4da80ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da8156750 0x7f4da80ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 -- 172.21.15.136:0/4246516579 >> v1:172.21.15.136:6789/0 conn(0x7f4da80cf9d0 legacy=0x7f4da806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:47.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.806+0000 7f4da5d9b640 1 -- 172.21.15.136:0/4246516579 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da815c990 con 0x7f4da80ca7d0 2024-04-25T11:35:47.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4da5d9b640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80ca7d0 0x7f4da80cf490 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009870 tx=0x7f4d9c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692df6986c7b05b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:47.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4da559a640 1 -- 172.21.15.136:0/4246516579 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0b2070 con 0x7f4da80ca7d0 2024-04-25T11:35:47.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4da559a640 1 -- 172.21.15.136:0/4246516579 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d9c0a9070 con 0x7f4da80ca7d0 2024-04-25T11:35:47.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4da559a640 1 -- 172.21.15.136:0/4246516579 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0ad880 con 0x7f4da80ca7d0 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- 172.21.15.136:0/4246516579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80ca7d0 msgr2=0x7f4da80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80ca7d0 0x7f4da80cf490 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009870 tx=0x7f4d9c01c560 comp rx=0 tx=0).stop 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- 172.21.15.136:0/4246516579 shutdown_connections 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80ca7d0 0x7f4da80cf490 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 --2- 172.21.15.136:0/4246516579 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da8156750 0x7f4da80ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- 172.21.15.136:0/4246516579 >> 172.21.15.136:0/4246516579 conn(0x7f4da8070f70 msgr2=0x7f4da8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- 172.21.15.136:0/4246516579 shutdown_connections 2024-04-25T11:35:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- 172.21.15.136:0/4246516579 wait complete. 2024-04-25T11:35:47.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 Processor -- start 2024-04-25T11:35:47.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.810+0000 7f4daccc4640 1 -- start start 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da80ca7d0 0x7f4da8154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4da80cdd70 0x7f4da8146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4da8169440 con 0x7f4da80cf9d0 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4da81692c0 con 0x7f4da80cdd70 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4da81695c0 con 0x7f4da80ca7d0 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34800/0 (socket says 172.21.15.136:34800) 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 -- 172.21.15.136:0/1272616036 learned_addr learned my addr 172.21.15.136:0/1272616036 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 -- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da80ca7d0 msgr2=0x7f4da8154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da80ca7d0 0x7f4da8154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:47.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 -- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4da80cdd70 msgr2=0x7f4da8146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4da80cdd70 0x7f4da8146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 -- 172.21.15.136:0/1272616036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da8155120 con 0x7f4da80cf9d0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4da5d9b640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0b0040 tx=0x7f4d9c019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0b2040 con 0x7f4da80cf9d0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d9c0b6650 con 0x7f4da80cf9d0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da81f5ca0 con 0x7f4da80cf9d0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0b6a40 con 0x7f4da80cf9d0 2024-04-25T11:35:47.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da81f6260 con 0x7f4da80cf9d0 2024-04-25T11:35:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.814+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da81f5a10 con 0x7f4da80cf9d0 2024-04-25T11:35:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.818+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4d9c0ab070 con 0x7f4da80cf9d0 2024-04-25T11:35:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.818+0000 7f4d96ffd640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 0x7f4d88071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.818+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f4d9c129430 con 0x7f4da80cf9d0 2024-04-25T11:35:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.818+0000 7f4da659c640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 0x7f4d88071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:47.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.818+0000 7f4da659c640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 0x7f4d88071910 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b850 tx=0x7f4d98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:47.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:47.822+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4d9c016150 con 0x7f4da80cf9d0 2024-04-25T11:35:48.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4da806a350 con 0x7f4da80cf9d0 2024-04-25T11:35:48.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4d96ffd640 1 -- 172.21.15.136:0/1272616036 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7f4d9c0f5de0 con 0x7f4da80cf9d0 2024-04-25T11:35:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:48.059 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:48.059 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:48.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 msgr2=0x7f4d88071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 0x7f4d88071910 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b850 tx=0x7f4d98008040 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 msgr2=0x7f4da8145dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0b0040 tx=0x7f4d9c019300 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 shutdown_connections 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4da80cdd70 0x7f4da8146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da80cf9d0 0x7f4da8145dd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4d8806f400 0x7f4d88071910 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 --2- 172.21.15.136:0/1272616036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4da80ca7d0 0x7f4da8154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 >> 172.21.15.136:0/1272616036 conn(0x7f4da8070f70 msgr2=0x7f4da8153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 shutdown_connections 2024-04-25T11:35:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:48.074+0000 7f4daccc4640 1 -- 172.21.15.136:0/1272616036 wait complete. 2024-04-25T11:35:48.079 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:50.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:50.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 Processor -- start 2024-04-25T11:35:50.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 -- start start 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc0c79e0 0x7fe0dc0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe0dc15a3d0 0x7fe0dc15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc0c9af0 con 0x7fe0dc0c96e0 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe0dc06cc10 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0e115a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe0dc0cb220 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0db59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe0dc15a3d0 0x7fe0dc15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0dad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe0dc0c96e0 0x7fe0dc1540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44832/0 (socket says 172.21.15.136:44832) 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0dad9d640 1 -- 172.21.15.136:0/1451565511 learned_addr learned my addr 172.21.15.136:0/1451565511 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0da59c640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc0c79e0 0x7fe0dc0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0db59e640 1 -- 172.21.15.136:0/1451565511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc0c79e0 msgr2=0x7fe0dc0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0db59e640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc0c79e0 0x7fe0dc0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0db59e640 1 -- 172.21.15.136:0/1451565511 >> v1:172.21.15.136:6789/0 conn(0x7fe0dc0c96e0 legacy=0x7fe0dc1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.266+0000 7fe0db59e640 1 -- 172.21.15.136:0/1451565511 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc0cb500 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0db59e640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc15a3d0 0x7fe0dc15a7e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009d90 tx=0x7fe0d401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd1a3b39b5ecd3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:50.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0d9d9b640 1 -- 172.21.15.136:0/1451565511 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40b2070 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0d9d9b640 1 -- 172.21.15.136:0/1451565511 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d40a5830 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0d9d9b640 1 -- 172.21.15.136:0/1451565511 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40a5c00 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 -- 172.21.15.136:0/1451565511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc15a3d0 msgr2=0x7fe0dc15a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc15a3d0 0x7fe0dc15a7e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009d90 tx=0x7fe0d401c970 comp rx=0 tx=0).stop 2024-04-25T11:35:50.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 -- 172.21.15.136:0/1451565511 shutdown_connections 2024-04-25T11:35:50.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc15a3d0 0x7fe0dc15a7e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 --2- 172.21.15.136:0/1451565511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc0c79e0 0x7fe0dc0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 -- 172.21.15.136:0/1451565511 >> 172.21.15.136:0/1451565511 conn(0x7fe0dc06f9d0 msgr2=0x7fe0dc06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:50.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 -- 172.21.15.136:0/1451565511 shutdown_connections 2024-04-25T11:35:50.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.270+0000 7fe0e115a640 1 -- 172.21.15.136:0/1451565511 wait complete. 2024-04-25T11:35:50.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 Processor -- start 2024-04-25T11:35:50.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 -- start start 2024-04-25T11:35:50.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0dc0c96e0 0x7fe0dc14e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc15a3d0 0x7fe0dc14ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc0ced40 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc06d440 con 0x7fe0dc0c96e0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0e115a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe0dc163050 con 0x7fe0dc15a3d0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34812/0 (socket says 172.21.15.136:34812) 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 -- 172.21.15.136:0/3619554699 learned_addr learned my addr 172.21.15.136:0/3619554699 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0db59e640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc15a3d0 0x7fe0dc14ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0da59c640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0dc0c96e0 0x7fe0dc14e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 -- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc15a3d0 msgr2=0x7fe0dc14ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc15a3d0 0x7fe0dc14ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 -- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0dc0c96e0 msgr2=0x7fe0dc14e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0dc0c96e0 0x7fe0dc14e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.274+0000 7fe0dad9d640 1 -- 172.21.15.136:0/3619554699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc164310 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0dad9d640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00c370 tx=0x7fe0cc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:50.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc017070 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0cc005cf0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0dc1645f0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc0072f0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0dc0c5ab0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0dc06a350 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe0cc01a070 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 0x7fe0c8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:50.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fe0cc08eac0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.278+0000 7fe0da59c640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 0x7fe0c8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:50.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.282+0000 7fe0da59c640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 0x7fe0c8071940 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0c54b0 tx=0x7fe0d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:50.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.286+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe0cc05ae20 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.502+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0dc0c55b0 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.502+0000 7fe0c37fe640 1 -- 172.21.15.136:0/3619554699 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7fe0cc05f120 con 0x7fe0dc0c79e0 2024-04-25T11:35:50.488 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:50.488 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:50.489 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 msgr2=0x7fe0c8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 0x7fe0c8071940 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0c54b0 tx=0x7fe0d0008040 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 msgr2=0x7fe0dc145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00c370 tx=0x7fe0cc00c830 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 shutdown_connections 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe0c806f430 0x7fe0c8071940 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0dc15a3d0 0x7fe0dc14ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0dc0c96e0 0x7fe0dc14e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 --2- 172.21.15.136:0/3619554699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0dc0c79e0 0x7fe0dc145d90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 >> 172.21.15.136:0/3619554699 conn(0x7fe0dc06f9d0 msgr2=0x7fe0dc153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 shutdown_connections 2024-04-25T11:35:50.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:50.506+0000 7fe0e115a640 1 -- 172.21.15.136:0/3619554699 wait complete. 2024-04-25T11:35:50.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:52.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:52.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 Processor -- start 2024-04-25T11:35:52.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 -- start start 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 0x7f51fc0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51fc0cbad0 con 0x7f51fc0ca010 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f51fc0c8550 con 0x7f51fc0ce260 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f520275e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc0c7c20 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 0x7f51fc0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f5200cfb640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f51fc0ca010 0x7f51fc0ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:33782/0 (socket says 172.21.15.136:33782) 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:38640/0 (socket says 172.21.15.136:38640) 2024-04-25T11:35:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f5200cfb640 1 -- 172.21.15.136:0/2645833616 learned_addr learned my addr 172.21.15.136:0/2645833616 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:35:52.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 -- 172.21.15.136:0/2645833616 >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 msgr2=0x7f51fc0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:52.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 --2- 172.21.15.136:0/2645833616 >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 0x7f51fc0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 -- 172.21.15.136:0/2645833616 >> v1:172.21.15.157:6789/0 conn(0x7f51fc0ca010 legacy=0x7f51fc0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fbfff640 1 -- 172.21.15.136:0/2645833616 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc0c7f00 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.706+0000 7f51fb7fe640 1 --2- 172.21.15.136:0/2645833616 >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 0x7f51fc0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:35:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f51fbfff640 1 --2- 172.21.15.136:0/2645833616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0c86d0 tx=0x7f51ec01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69233f796dd7481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f51faffd640 1 -- 172.21.15.136:0/2645833616 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b5070 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f51faffd640 1 -- 172.21.15.136:0/2645833616 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec0a8990 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f51faffd640 1 -- 172.21.15.136:0/2645833616 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0a8d60 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 -- 172.21.15.136:0/2645833616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 msgr2=0x7f51fc14f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 --2- 172.21.15.136:0/2645833616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0c86d0 tx=0x7f51ec01ec90 comp rx=0 tx=0).stop 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 -- 172.21.15.136:0/2645833616 shutdown_connections 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 --2- 172.21.15.136:0/2645833616 >> v2:172.21.15.157:3300/0 conn(0x7f51fc0ce260 0x7f51fc0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 --2- 172.21.15.136:0/2645833616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0c8cd0 0x7f51fc14f680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 -- 172.21.15.136:0/2645833616 >> 172.21.15.136:0/2645833616 conn(0x7f51fc06f9d0 msgr2=0x7f51fc06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 -- 172.21.15.136:0/2645833616 shutdown_connections 2024-04-25T11:35:52.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.710+0000 7f520275e640 1 -- 172.21.15.136:0/2645833616 wait complete. 2024-04-25T11:35:52.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.714+0000 7f520275e640 1 Processor -- start 2024-04-25T11:35:52.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.714+0000 7f520275e640 1 -- start start 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51fc0ca010 0x7f51fc143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0ce260 0x7f51fc1380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc0cb720 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc14fbc0 con 0x7f51fc0ca010 2024-04-25T11:35:52.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc1f1ca0 con 0x7f51fc0ce260 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34820/0 (socket says 172.21.15.136:34820) 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 -- 172.21.15.136:0/3490334938 learned_addr learned my addr 172.21.15.136:0/3490334938 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f5200cfb640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0ce260 0x7f51fc1380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fb7fe640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51fc0ca010 0x7f51fc143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 -- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0ce260 msgr2=0x7f51fc1380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0ce260 0x7f51fc1380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 -- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51fc0ca010 msgr2=0x7f51fc143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51fc0ca010 0x7f51fc143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 -- 172.21.15.136:0/3490334938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc1f1a90 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51fbfff640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0b3040 tx=0x7f51ec01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0bf070 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec0b0640 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc1436c0 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b5040 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc142980 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51fc06a350 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51ec0c2070 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.718+0000 7f51f8ff9640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 0x7f51e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:52.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.722+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f51ec12bfa0 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.722+0000 7f51fb7fe640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 0x7f51e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:52.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.722+0000 7f51fb7fe640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 0x7f51e8071910 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0c60b0 tx=0x7f51f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:52.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.726+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f51ec0187f0 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.954+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51fc1645e0 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f51f8ff9640 1 -- 172.21.15.136:0/3490334938 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7f51ec0f88c0 con 0x7f51fc0c8cd0 2024-04-25T11:35:52.943 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:52.943 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:52.943 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:52.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 msgr2=0x7f51e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 0x7f51e8071910 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0c60b0 tx=0x7f51f0008040 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 msgr2=0x7f51fc141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0b3040 tx=0x7f51ec01ba30 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 shutdown_connections 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51e806f400 0x7f51e8071910 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51fc0ce260 0x7f51fc1380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51fc0ca010 0x7f51fc143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 --2- 172.21.15.136:0/3490334938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51fc0c8cd0 0x7f51fc141a50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 >> 172.21.15.136:0/3490334938 conn(0x7f51fc06f9d0 msgr2=0x7f51fc14c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 shutdown_connections 2024-04-25T11:35:52.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:52.958+0000 7f520275e640 1 -- 172.21.15.136:0/3490334938 wait complete. 2024-04-25T11:35:52.965 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:53.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:35:53.718 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:35:54.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:55.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.142+0000 7f34cc6df640 1 Processor -- start 2024-04-25T11:35:55.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.142+0000 7f34cc6df640 1 -- start start 2024-04-25T11:35:55.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cc6df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c415a5f0 0x7f34c40c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cc6df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34c415aa00 0x7f34c40c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cc6df640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34c40c3880 con 0x7f34c4154060 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cc6df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f34c40c8fd0 con 0x7f34c415aa00 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cc6df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34c406d9b0 con 0x7f34c415a5f0 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34ca47b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f34c4154060 0x7f34c40c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33288/0 (socket says 172.21.15.136:33288) 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34ca47b640 1 -- 172.21.15.136:0/2249600218 learned_addr learned my addr 172.21.15.136:0/2249600218 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054790998 0 0) 0x7f34c40c3880 con 0x7f34c4154060 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34b0003530 con 0x7f34c4154060 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c415a5f0 0x7f34c40c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cac7c640 1 --2- 172.21.15.136:0/2249600218 >> v2:172.21.15.157:3300/0 conn(0x7f34c415aa00 0x7f34c40c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 941454490 0 0) 0x7f34b0003530 con 0x7f34c4154060 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c415a5f0 msgr2=0x7f34c40c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c415a5f0 0x7f34c40c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 >> v2:172.21.15.157:3300/0 conn(0x7f34c415aa00 msgr2=0x7f34c40c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 --2- 172.21.15.136:0/2249600218 >> v2:172.21.15.157:3300/0 conn(0x7f34c415aa00 0x7f34c40c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c4154470 con 0x7f34c4154060 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f34c00016a0 con 0x7f34c4154060 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 >> v1:172.21.15.136:6789/0 conn(0x7f34c4154060 legacy=0x7f34c40c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b00039c0 0x7f34b0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34b0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34b0005160 0x7f34b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f34c40c8fd0 con 0x7f34b0004810 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34c406d9b0 con 0x7f34b00039c0 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34b000c050 con 0x7f34b0005160 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c415a5f0 0x7f34c40c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34b0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34ca47b640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b00039c0 0x7f34b0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34cac7c640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34b0005160 0x7f34b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 -- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34b0005160 msgr2=0x7f34b000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:55.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34b0005160 0x7f34b000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 -- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b00039c0 msgr2=0x7f34b0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:55.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b00039c0 0x7f34b0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.146+0000 7f34c9c7a640 1 -- 172.21.15.136:0/2249600218 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b0003530 con 0x7f34b0004810 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34b0004c20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f34c4073070 tx=0x7f34b401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802fecbc94013305 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b400f040 con 0x7f34b0004810 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b40b0070 con 0x7f34b0004810 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34c9479640 1 -- 172.21.15.136:0/2249600218 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40ab820 con 0x7f34b0004810 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 -- 172.21.15.136:0/2249600218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 msgr2=0x7f34b0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 --2- 172.21.15.136:0/2249600218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34b0004c20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f34c4073070 tx=0x7f34b401c3b0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cac7c640 1 -- 172.21.15.136:0/2249600218 reap_dead start 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 -- 172.21.15.136:0/2249600218 shutdown_connections 2024-04-25T11:35:55.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 -- 172.21.15.136:0/2249600218 >> 172.21.15.136:0/2249600218 conn(0x7f34c40ca5a0 msgr2=0x7f34c40ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:55.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 -- 172.21.15.136:0/2249600218 shutdown_connections 2024-04-25T11:35:55.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 -- 172.21.15.136:0/2249600218 wait complete. 2024-04-25T11:35:55.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.150+0000 7f34cc6df640 1 Processor -- start 2024-04-25T11:35:55.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- start start 2024-04-25T11:35:55.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34c41e5c50 0x7f34c41f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c414e120 0x7f34c414e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f34c415a0d0 con 0x7f34b0004810 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34c406df30 con 0x7f34c41e5c50 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34c4159d80 con 0x7f34c414e120 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47216/0 (socket says 172.21.15.136:47216) 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 -- 172.21.15.136:0/424830590 learned_addr learned my addr 172.21.15.136:0/424830590 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34c41e5c50 0x7f34c41f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cac7c640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c414e120 0x7f34c414e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 -- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c414e120 msgr2=0x7f34c414e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c414e120 0x7f34c414e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 -- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34c41e5c50 msgr2=0x7f34c41f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34c41e5c50 0x7f34c41f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 -- 172.21.15.136:0/424830590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c414dea0 con 0x7f34b0004810 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34ca47b640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f34c0001230 tx=0x7f34c000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c000c6c0 con 0x7f34b0004810 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0018070 con 0x7f34b0004810 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c41f4540 con 0x7f34b0004810 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c000cac0 con 0x7f34b0004810 2024-04-25T11:35:55.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c4148f70 con 0x7f34b0004810 2024-04-25T11:35:55.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.154+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c406a350 con 0x7f34b0004810 2024-04-25T11:35:55.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.158+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f34c000cc60 con 0x7f34b0004810 2024-04-25T11:35:55.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.158+0000 7f34baffd640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 0x7f34a8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:55.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.158+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7f34c008f770 con 0x7f34b0004810 2024-04-25T11:35:55.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.158+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 0x7f34a8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:55.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.158+0000 7f34c9c7a640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 0x7f34a8071940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f34b40af000 tx=0x7f34b40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:55.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.162+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f34c41f4540 con 0x7f34b0004810 2024-04-25T11:35:55.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.398+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34c40c85e0 con 0x7f34b0004810 2024-04-25T11:35:55.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.398+0000 7f34baffd640 1 -- 172.21.15.136:0/424830590 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7f34c005c010 con 0x7f34b0004810 2024-04-25T11:35:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:55.384 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:55.384 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 msgr2=0x7f34a8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 0x7f34a8071940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f34b40af000 tx=0x7f34b40046d0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 msgr2=0x7f34c4074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f34c0001230 tx=0x7f34c000eba0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 shutdown_connections 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f430 0x7f34a8071940 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34c414e120 0x7f34c414e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34c41e5c50 0x7f34c41f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 --2- 172.21.15.136:0/424830590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34b0004810 0x7f34c4074c00 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 >> 172.21.15.136:0/424830590 conn(0x7f34c40ca5a0 msgr2=0x7f34c41578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:55.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 shutdown_connections 2024-04-25T11:35:55.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:55.402+0000 7f34cc6df640 1 -- 172.21.15.136:0/424830590 wait complete. 2024-04-25T11:35:55.407 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:57.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:57.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.578+0000 7fd6b673b640 1 Processor -- start 2024-04-25T11:35:57.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.578+0000 7fd6b673b640 1 -- start start 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b673b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6b00c8cd0 0x7fd6b014f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b673b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6b00ca720 0x7fd6b00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b673b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6b00cbad0 con 0x7fd6b00ce260 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b673b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd6b00c8550 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b673b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd6b00c7c20 con 0x7fd6b00ca720 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6b00c8cd0 0x7fd6b014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6b00c8cd0 0x7fd6b014f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47232/0 (socket says 172.21.15.136:47232) 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 -- 172.21.15.136:0/856962010 learned_addr learned my addr 172.21.15.136:0/856962010 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6b4cd8640 1 --2- 172.21.15.136:0/856962010 >> v2:172.21.15.157:3300/0 conn(0x7fd6b00ca720 0x7fd6b00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 -- 172.21.15.136:0/856962010 >> v2:172.21.15.157:3300/0 conn(0x7fd6b00ca720 msgr2=0x7fd6b00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 --2- 172.21.15.136:0/856962010 >> v2:172.21.15.157:3300/0 conn(0x7fd6b00ca720 0x7fd6b00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 -- 172.21.15.136:0/856962010 >> v1:172.21.15.157:6789/0 conn(0x7fd6b00ce260 legacy=0x7fd6b00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:57.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.582+0000 7fd6affff640 1 -- 172.21.15.136:0/856962010 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b00c7f00 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6affff640 1 --2- 172.21.15.136:0/856962010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b014f680 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00c86d0 tx=0x7fd6a0020ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce6095691479e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:57.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6aeffd640 1 -- 172.21.15.136:0/856962010 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00b7070 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6aeffd640 1 -- 172.21.15.136:0/856962010 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a00aabe0 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6aeffd640 1 -- 172.21.15.136:0/856962010 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00b2440 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- 172.21.15.136:0/856962010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00c8cd0 msgr2=0x7fd6b014f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 --2- 172.21.15.136:0/856962010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b014f680 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00c86d0 tx=0x7fd6a0020ee0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- 172.21.15.136:0/856962010 shutdown_connections 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 --2- 172.21.15.136:0/856962010 >> v2:172.21.15.157:3300/0 conn(0x7fd6b00ca720 0x7fd6b00ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 --2- 172.21.15.136:0/856962010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b014f680 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- 172.21.15.136:0/856962010 >> 172.21.15.136:0/856962010 conn(0x7fd6b006f9d0 msgr2=0x7fd6b006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- 172.21.15.136:0/856962010 shutdown_connections 2024-04-25T11:35:57.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- 172.21.15.136:0/856962010 wait complete. 2024-04-25T11:35:57.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 Processor -- start 2024-04-25T11:35:57.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.586+0000 7fd6b673b640 1 -- start start 2024-04-25T11:35:57.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b0145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6b00ce260 0x7fd6b006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b00cb720 con 0x7fd6b00ca720 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b014fbc0 con 0x7fd6b00c8cd0 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b01ef660 con 0x7fd6b00ce260 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47244/0 (socket says 172.21.15.136:47244) 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 -- 172.21.15.136:0/1926135374 learned_addr learned my addr 172.21.15.136:0/1926135374 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b4cd8640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6b00ce260 0x7fd6b006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6affff640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b0145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 -- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6b00ce260 msgr2=0x7fd6b006dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6b00ce260 0x7fd6b006dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 -- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6b00c8cd0 msgr2=0x7fd6b0145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b0145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 -- 172.21.15.136:0/1926135374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b0145860 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6af7fe640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400b3c0 tx=0x7fd6a400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4011830 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a4016070 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4011c30 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b01f1870 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b0153420 con 0x7fd6b00ca720 2024-04-25T11:35:57.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b006a350 con 0x7fd6b00ca720 2024-04-25T11:35:57.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.590+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6a401b460 con 0x7fd6b00ca720 2024-04-25T11:35:57.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.594+0000 7fd6acff9640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 0x7fd69c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:57.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.594+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7337+0+0 (secure 0 0 0) 0x7fd6a408f3a0 con 0x7fd6b00ca720 2024-04-25T11:35:57.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.594+0000 7fd6affff640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 0x7fd69c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:57.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.594+0000 7fd6affff640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 0x7fd69c071940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00b4000 tx=0x7fd6a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:57.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.598+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6a4020020 con 0x7fd6b00ca720 2024-04-25T11:35:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.818+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6b0156790 con 0x7fd6b00ca720 2024-04-25T11:35:57.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6acff9640 1 -- 172.21.15.136:0/1926135374 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6931 (secure 0 0 0) 0x7fd6a405bc40 con 0x7fd6b00ca720 2024-04-25T11:35:57.807 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:35:57.807 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5249,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:33:49.323572+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5238":{"gid":5238,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6835/3406304193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3406304193},{"type":"v1","addr":"172.21.15.136:6835","nonce":3406304193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5238]},"id":1}]} 2024-04-25T11:35:57.807 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 61 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 msgr2=0x7fd69c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 0x7fd69c071940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00b4000 tx=0x7fd6a0008040 comp rx=0 tx=0).stop 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 msgr2=0x7fd6b014a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400b3c0 tx=0x7fd6a400b880 comp rx=0 tx=0).stop 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 shutdown_connections 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd69c06f430 0x7fd69c071940 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6b00ce260 0x7fd6b006dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b00ca720 0x7fd6b014a780 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 --2- 172.21.15.136:0/1926135374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6b00c8cd0 0x7fd6b0145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 >> 172.21.15.136:0/1926135374 conn(0x7fd6b006f9d0 msgr2=0x7fd6b0144250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 shutdown_connections 2024-04-25T11:35:57.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:35:57.822+0000 7fd6b673b640 1 -- 172.21.15.136:0/1926135374 wait complete. 2024-04-25T11:35:57.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T11:35:59.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:35:59.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 Processor -- start 2024-04-25T11:35:59.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 -- start start 2024-04-25T11:35:59.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f215815c070 0x7f215806d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:59.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f215815c480 0x7f215806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:59.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21580b4480 con 0x7f21580b5810 2024-04-25T11:35:59.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f21580b4a90 con 0x7f215815c480 2024-04-25T11:35:59.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.002+0000 7f215de3c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f215815c990 con 0x7f215815c070 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f214effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f215815c070 0x7f215806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f21577fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f21580b5810 0x7f21580b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:42278/0 (socket says 172.21.15.136:42278) 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f21577fe640 1 -- 172.21.15.136:0/238768431 learned_addr learned my addr 172.21.15.136:0/238768431 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 --2- 172.21.15.136:0/238768431 >> v2:172.21.15.157:3300/0 conn(0x7f215815c480 0x7f215806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 -- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f215815c070 msgr2=0x7f215806d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 --2- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f215815c070 0x7f215806d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 -- 172.21.15.136:0/238768431 >> v1:172.21.15.157:6789/0 conn(0x7f21580b5810 legacy=0x7f21580b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:59.991 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 -- 172.21.15.136:0/238768431 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215815cc70 con 0x7f215815c480 2024-04-25T11:35:59.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2157fff640 1 --2- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f215806da50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2150009d90 tx=0x7f215001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b79120bb53b6322f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:35:59.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2156ffd640 1 -- 172.21.15.136:0/238768431 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21500b4070 con 0x7f215815c480 2024-04-25T11:35:59.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.006+0000 7f2156ffd640 1 -- 172.21.15.136:0/238768431 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21500af600 con 0x7f215815c480 2024-04-25T11:35:59.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f2156ffd640 1 -- 172.21.15.136:0/238768431 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21500af9d0 con 0x7f215815c480 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 msgr2=0x7f215806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 --2- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f215806da50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2150009d90 tx=0x7f215001f150 comp rx=0 tx=0).stop 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- 172.21.15.136:0/238768431 shutdown_connections 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 --2- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f215806da50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 --2- 172.21.15.136:0/238768431 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f215815c070 0x7f215806d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- 172.21.15.136:0/238768431 >> 172.21.15.136:0/238768431 conn(0x7f21580743a0 msgr2=0x7f21580d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- 172.21.15.136:0/238768431 shutdown_connections 2024-04-25T11:35:59.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- 172.21.15.136:0/238768431 wait complete. 2024-04-25T11:35:59.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 Processor -- start 2024-04-25T11:35:59.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.010+0000 7f215de3c640 1 -- start start 2024-04-25T11:35:59.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21580b5810 0x7f21581e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:59.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:35:59.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f2158153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f215806e030 con 0x7f215815c070 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f215815cf50 con 0x7f215815c480 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f21580cbee0 con 0x7f21580b5810 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47254/0 (socket says 172.21.15.136:47254) 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 -- 172.21.15.136:0/4119436963 learned_addr learned my addr 172.21.15.136:0/4119436963 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f21577fe640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21580b5810 0x7f21581e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f2157fff640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f2158153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 -- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21580b5810 msgr2=0x7f21581e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21580b5810 0x7f21581e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 -- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 msgr2=0x7f2158153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f2158153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 -- 172.21.15.136:0/4119436963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215815ed70 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f214effd640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f214000e8a0 tx=0x7f214000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140017070 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f214000c940 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214000ccf0 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215815f030 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21580c5210 con 0x7f215815c070 2024-04-25T11:36:00.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.014+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215806a350 con 0x7f215815c070 2024-04-25T11:36:00.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.018+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f214001a070 con 0x7f215815c070 2024-04-25T11:36:00.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.018+0000 7f2154ff9640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 0x7f2138071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.003 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.018+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f214008fe80 con 0x7f215815c070 2024-04-25T11:36:00.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.018+0000 7f21577fe640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 0x7f2138071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.018+0000 7f21577fe640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 0x7f2138071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2148000f70 tx=0x7f2148007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:00.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.022+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f214005c1e0 con 0x7f215815c070 2024-04-25T11:36:00.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.286+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f215806e510 con 0x7f215815c070 2024-04-25T11:36:00.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.286+0000 7f2154ff9640 1 -- 172.21.15.136:0/4119436963 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6098 (secure 0 0 0) 0x7f21400604e0 con 0x7f215815c070 2024-04-25T11:36:00.273 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:00.273 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:35:58.956475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:00.274 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 62 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 msgr2=0x7f2138071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 0x7f2138071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2148000f70 tx=0x7f2148007e90 comp rx=0 tx=0).stop 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 msgr2=0x7f2158153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f214000e8a0 tx=0x7f214000eca0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 shutdown_connections 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f215815c480 0x7f2158153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f215815c070 0x7f2158153820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f213806f430 0x7f2138071940 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 --2- 172.21.15.136:0/4119436963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21580b5810 0x7f21581e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 >> 172.21.15.136:0/4119436963 conn(0x7f21580743a0 msgr2=0x7f2158155ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:00.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 shutdown_connections 2024-04-25T11:36:00.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.290+0000 7f215de3c640 1 -- 172.21.15.136:0/4119436963 wait complete. 2024-04-25T11:36:00.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-25T11:36:00.296 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:36:00.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:00.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 Processor -- start 2024-04-25T11:36:00.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 -- start start 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8bc0ca290 0x7fc8bc0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 0x7fc8bc15a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8bc15d750 con 0x7fc8bc0b5060 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc8bc15da30 con 0x7fc8bc0ca290 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c31bb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc8bc15dd10 con 0x7fc8bc15a430 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8bbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8bc0ca290 0x7fc8bc0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c0f57640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc8bc0b5060 0x7fc8bc0c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:33296/0 (socket says 172.21.15.136:33296) 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c0f57640 1 -- 172.21.15.136:0/188489725 learned_addr learned my addr 172.21.15.136:0/188489725 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c1758640 1 --2- 172.21.15.136:0/188489725 >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 0x7fc8bc15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8bbfff640 1 -- 172.21.15.136:0/188489725 >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 msgr2=0x7fc8bc15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8bbfff640 1 --2- 172.21.15.136:0/188489725 >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 0x7fc8bc15a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8bbfff640 1 -- 172.21.15.136:0/188489725 >> v1:172.21.15.136:6789/0 conn(0x7fc8bc0b5060 legacy=0x7fc8bc0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8bbfff640 1 -- 172.21.15.136:0/188489725 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc15dff0 con 0x7fc8bc0ca290 2024-04-25T11:36:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.458+0000 7fc8c1758640 1 --2- 172.21.15.136:0/188489725 >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 0x7fc8bc15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:00.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8bbfff640 1 --2- 172.21.15.136:0/188489725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc0ca6a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0b4fe0 tx=0x7fc8a801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b933f88d218744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:00.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8bb7fe640 1 -- 172.21.15.136:0/188489725 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80b5070 con 0x7fc8bc0ca290 2024-04-25T11:36:00.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8bb7fe640 1 -- 172.21.15.136:0/188489725 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a80a8970 con 0x7fc8bc0ca290 2024-04-25T11:36:00.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8bb7fe640 1 -- 172.21.15.136:0/188489725 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80a8d40 con 0x7fc8bc0ca290 2024-04-25T11:36:00.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 -- 172.21.15.136:0/188489725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 msgr2=0x7fc8bc0ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/188489725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc0ca6a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc0b4fe0 tx=0x7fc8a801ec70 comp rx=0 tx=0).stop 2024-04-25T11:36:00.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 -- 172.21.15.136:0/188489725 shutdown_connections 2024-04-25T11:36:00.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/188489725 >> v2:172.21.15.157:3300/0 conn(0x7fc8bc15a430 0x7fc8bc15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/188489725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc0ca6a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 -- 172.21.15.136:0/188489725 >> 172.21.15.136:0/188489725 conn(0x7fc8bc070f70 msgr2=0x7fc8bc068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 -- 172.21.15.136:0/188489725 shutdown_connections 2024-04-25T11:36:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.462+0000 7fc8c31bb640 1 -- 172.21.15.136:0/188489725 wait complete. 2024-04-25T11:36:00.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 Processor -- start 2024-04-25T11:36:00.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- start start 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 0x7fc8bc148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8bc15a430 0x7fc8bc1452d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc15ad80 con 0x7fc8bc0ca290 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc159f10 con 0x7fc8bc15a430 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8bc15d3f0 con 0x7fc8bc0b5060 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47276/0 (socket says 172.21.15.136:47276) 2024-04-25T11:36:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 -- 172.21.15.136:0/1837056895 learned_addr learned my addr 172.21.15.136:0/1837056895 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c1758640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8bc15a430 0x7fc8bc1452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c0f57640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 0x7fc8bc148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 -- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 msgr2=0x7fc8bc148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 0x7fc8bc148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 -- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8bc15a430 msgr2=0x7fc8bc1452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8bc15a430 0x7fc8bc1452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 -- 172.21.15.136:0/1837056895 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8bc14f6e0 con 0x7fc8bc0ca290 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c0f57640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 0x7fc8bc148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8bbfff640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80b3040 tx=0x7fc8a801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80bf070 con 0x7fc8bc0ca290 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a80b0800 con 0x7fc8bc0ca290 2024-04-25T11:36:00.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8bc0c6220 con 0x7fc8bc0ca290 2024-04-25T11:36:00.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80b5040 con 0x7fc8bc0ca290 2024-04-25T11:36:00.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.466+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8bc151f40 con 0x7fc8bc0ca290 2024-04-25T11:36:00.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8bc06a350 con 0x7fc8bc0ca290 2024-04-25T11:36:00.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc8a80c2070 con 0x7fc8bc0ca290 2024-04-25T11:36:00.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8b97fa640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 0x7fc8a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:00.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fc8a812c170 con 0x7fc8bc0ca290 2024-04-25T11:36:00.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8c0f57640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 0x7fc8a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:00.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.470+0000 7fc8c0f57640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 0x7fc8a0071910 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0007fd0 tx=0x7fc8b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:00.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.474+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc8a80187d0 con 0x7fc8bc0ca290 2024-04-25T11:36:00.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.734+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8bc14fc30 con 0x7fc8bc0ca290 2024-04-25T11:36:00.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.734+0000 7fc8b97fa640 1 -- 172.21.15.136:0/1837056895 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6098 (secure 0 0 0) 0x7fc8a80f8a40 con 0x7fc8bc0ca290 2024-04-25T11:36:00.721 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:00.721 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:35:58.956475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:00.721 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 62 2024-04-25T11:36:00.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 msgr2=0x7fc8a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 0x7fc8a0071910 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0007fd0 tx=0x7fc8b0009040 comp rx=0 tx=0).stop 2024-04-25T11:36:00.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 msgr2=0x7fc8bc144d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:00.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80b3040 tx=0x7fc8a801ba10 comp rx=0 tx=0).stop 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 shutdown_connections 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8bc15a430 0x7fc8bc1452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8bc0ca290 0x7fc8bc144d90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8a006f400 0x7fc8a0071910 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 --2- 172.21.15.136:0/1837056895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8bc0b5060 0x7fc8bc148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 >> 172.21.15.136:0/1837056895 conn(0x7fc8bc070f70 msgr2=0x7fc8bc0cb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 shutdown_connections 2024-04-25T11:36:00.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:00.738+0000 7fc8c31bb640 1 -- 172.21.15.136:0/1837056895 wait complete. 2024-04-25T11:36:00.747 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5249": { "addr": "172.21.15.157:6837/3576995854", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3576995854, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3576995854, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 62, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:35:58.956475+0000", "qdb_cluster": [ 5356, 5385, 5449, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 5249 } } } ], "standbys": [] } 2024-04-25T11:36:02.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:02.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 Processor -- start 2024-04-25T11:36:02.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 -- start start 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200ca010 0x7f8a2006cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8a2006d040 0x7f8a2015c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a200c8a20 con 0x7f8a200c9b50 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2006c670 con 0x7f8a2006d040 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a27415640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a200c8d00 con 0x7f8a200ca010 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8a2006d040 0x7f8a2015c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8a2006d040 0x7f8a2015c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47282/0 (socket says 172.21.15.136:47282) 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 -- 172.21.15.136:0/2112177862 learned_addr learned my addr 172.21.15.136:0/2112177862 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a251b1640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200ca010 0x7f8a2006cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 -- 172.21.15.136:0/2112177862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200ca010 msgr2=0x7f8a2006cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:02.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200ca010 0x7f8a2006cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:02.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.934+0000 7f8a259b2640 1 -- 172.21.15.136:0/2112177862 >> v1:172.21.15.157:6789/0 conn(0x7f8a200c9b50 legacy=0x7f8a2015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:02.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.938+0000 7f8a259b2640 1 -- 172.21.15.136:0/2112177862 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a20074d10 con 0x7f8a2006d040 2024-04-25T11:36:02.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.938+0000 7f8a259b2640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a2006d040 0x7f8a2015c6f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009e50 tx=0x7f8a1801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cb081e9dd8cda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:02.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.938+0000 7f8a17fff640 1 -- 172.21.15.136:0/2112177862 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180a5700 con 0x7f8a2006d040 2024-04-25T11:36:02.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.938+0000 7f8a17fff640 1 -- 172.21.15.136:0/2112177862 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a180a58a0 con 0x7f8a2006d040 2024-04-25T11:36:02.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.938+0000 7f8a17fff640 1 -- 172.21.15.136:0/2112177862 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180a5c70 con 0x7f8a2006d040 2024-04-25T11:36:02.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- 172.21.15.136:0/2112177862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a2006d040 msgr2=0x7f8a2015c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:02.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a2006d040 0x7f8a2015c6f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009e50 tx=0x7f8a1801ca30 comp rx=0 tx=0).stop 2024-04-25T11:36:02.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- 172.21.15.136:0/2112177862 shutdown_connections 2024-04-25T11:36:02.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a2006d040 0x7f8a2015c6f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:02.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 --2- 172.21.15.136:0/2112177862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200ca010 0x7f8a2006cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:02.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- 172.21.15.136:0/2112177862 >> 172.21.15.136:0/2112177862 conn(0x7f8a200ca5a0 msgr2=0x7f8a200ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:02.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- 172.21.15.136:0/2112177862 shutdown_connections 2024-04-25T11:36:02.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- 172.21.15.136:0/2112177862 wait complete. 2024-04-25T11:36:02.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 Processor -- start 2024-04-25T11:36:02.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.942+0000 7f8a27415640 1 -- start start 2024-04-25T11:36:02.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2006d040 0x7f8a201f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200c9b50 0x7f8a2015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2015cdb0 con 0x7f8a200ca010 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2015cc30 con 0x7f8a2006d040 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a20169140 con 0x7f8a200c9b50 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47284/0 (socket says 172.21.15.136:47284) 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 -- 172.21.15.136:0/2433549145 learned_addr learned my addr 172.21.15.136:0/2433549145 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a249b0640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200c9b50 0x7f8a2015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a251b1640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2006d040 0x7f8a201f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 -- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200c9b50 msgr2=0x7f8a2015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200c9b50 0x7f8a2015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 -- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2006d040 msgr2=0x7f8a201f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:02.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2006d040 0x7f8a201f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 -- 172.21.15.136:0/2433549145 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a201f94e0 con 0x7f8a200ca010 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a259b2640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8a1801a660 tx=0x7f8a180197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180bc020 con 0x7f8a200ca010 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a180b8670 con 0x7f8a200ca010 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a201f87e0 con 0x7f8a200ca010 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180b8a20 con 0x7f8a200ca010 2024-04-25T11:36:02.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a20154250 con 0x7f8a200ca010 2024-04-25T11:36:02.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2006a350 con 0x7f8a200ca010 2024-04-25T11:36:02.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8a180bf020 con 0x7f8a200ca010 2024-04-25T11:36:02.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 0x7f8a04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:02.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.946+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f8a1812a890 con 0x7f8a200ca010 2024-04-25T11:36:02.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.950+0000 7f8a251b1640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 0x7f8a04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:02.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.950+0000 7f8a251b1640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 0x7f8a04071910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00b930 tx=0x7f8a1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:02.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:02.954+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8a180c6030 con 0x7f8a200ca010 2024-04-25T11:36:03.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a200c7ab0 con 0x7f8a200ca010 2024-04-25T11:36:03.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a15ffb640 1 -- 172.21.15.136:0/2433549145 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6098 (secure 0 0 0) 0x7f8a18016620 con 0x7f8a200ca010 2024-04-25T11:36:03.171 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:03.171 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:35:58.956475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:03.171 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 62 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 msgr2=0x7f8a04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 0x7f8a04071910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00b930 tx=0x7f8a1c008040 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 msgr2=0x7f8a201f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8a1801a660 tx=0x7f8a180197d0 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 shutdown_connections 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a200ca010 0x7f8a201f58a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8a0406f400 0x7f8a04071910 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a200c9b50 0x7f8a2015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 --2- 172.21.15.136:0/2433549145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2006d040 0x7f8a201f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 >> 172.21.15.136:0/2433549145 conn(0x7f8a200ca5a0 msgr2=0x7f8a200cd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:03.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 shutdown_connections 2024-04-25T11:36:03.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:03.186+0000 7f8a27415640 1 -- 172.21.15.136:0/2433549145 wait complete. 2024-04-25T11:36:03.196 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:05.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:05.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 Processor -- start 2024-04-25T11:36:05.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 -- start start 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fddac15c0d0 0x7fddac0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fddac15ff60 0x7fddac0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddac15c6f0 con 0x7fddac160370 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fddac0c6310 con 0x7fddac15c0d0 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb32b9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fddac15c990 con 0x7fddac15ff60 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb1055640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fddac15c0d0 0x7fddac0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb1055640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fddac15c0d0 0x7fddac0cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51208/0 (socket says 172.21.15.136:51208) 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.370+0000 7fddb1055640 1 -- 172.21.15.136:0/1250419555 learned_addr learned my addr 172.21.15.136:0/1250419555 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb0854640 1 --2- 172.21.15.136:0/1250419555 >> v2:172.21.15.157:3300/0 conn(0x7fddac15ff60 0x7fddac0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb1055640 1 -- 172.21.15.136:0/1250419555 >> v2:172.21.15.157:3300/0 conn(0x7fddac15ff60 msgr2=0x7fddac0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:05.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb1055640 1 --2- 172.21.15.136:0/1250419555 >> v2:172.21.15.157:3300/0 conn(0x7fddac15ff60 0x7fddac0ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb1055640 1 -- 172.21.15.136:0/1250419555 >> v1:172.21.15.157:6789/0 conn(0x7fddac160370 legacy=0x7fddac071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:05.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb1055640 1 -- 172.21.15.136:0/1250419555 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddac15cc30 con 0x7fddac15c0d0 2024-04-25T11:36:05.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb1055640 1 --2- 172.21.15.136:0/1250419555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac15c0d0 0x7fddac0cb720 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fddac0caf60 tx=0x7fdda801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5addc7d4c15f2660 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:05.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fdda3fff640 1 -- 172.21.15.136:0/1250419555 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda80b5070 con 0x7fddac15c0d0 2024-04-25T11:36:05.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fdda3fff640 1 -- 172.21.15.136:0/1250419555 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda80a89d0 con 0x7fddac15c0d0 2024-04-25T11:36:05.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fdda3fff640 1 -- 172.21.15.136:0/1250419555 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda80a8da0 con 0x7fddac15c0d0 2024-04-25T11:36:05.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 -- 172.21.15.136:0/1250419555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac15c0d0 msgr2=0x7fddac0cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:05.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1250419555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac15c0d0 0x7fddac0cb720 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fddac0caf60 tx=0x7fdda801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 -- 172.21.15.136:0/1250419555 shutdown_connections 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1250419555 >> v2:172.21.15.157:3300/0 conn(0x7fddac15ff60 0x7fddac0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1250419555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac15c0d0 0x7fddac0cb720 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 -- 172.21.15.136:0/1250419555 >> 172.21.15.136:0/1250419555 conn(0x7fddac06f9d0 msgr2=0x7fddac06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 -- 172.21.15.136:0/1250419555 shutdown_connections 2024-04-25T11:36:05.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.374+0000 7fddb32b9640 1 -- 172.21.15.136:0/1250419555 wait complete. 2024-04-25T11:36:05.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 Processor -- start 2024-04-25T11:36:05.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 -- start start 2024-04-25T11:36:05.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 0x7fddac1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fddac15ff60 0x7fddac14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fddac0cbc60 con 0x7fddac160370 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fddac15fa40 con 0x7fddac15ff60 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb32b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fddac15cf10 con 0x7fddac15c0d0 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51210/0 (socket says 172.21.15.136:51210) 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 -- 172.21.15.136:0/1561007088 learned_addr learned my addr 172.21.15.136:0/1561007088 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb0854640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fddac15ff60 0x7fddac14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1055640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 0x7fddac1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 -- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 msgr2=0x7fddac1f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 0x7fddac1f6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 -- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fddac15ff60 msgr2=0x7fddac14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fddac15ff60 0x7fddac14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 -- 172.21.15.136:0/1561007088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddac06d0a0 con 0x7fddac160370 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1055640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 0x7fddac1f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:05.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fddb1856640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdda400ba10 tx=0x7fdda400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:05.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.378+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4019070 con 0x7fddac160370 2024-04-25T11:36:05.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda400cb40 con 0x7fddac160370 2024-04-25T11:36:05.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddac15eea0 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4014440 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddac1f6060 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddac1f5e50 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdda401c070 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fdda1ffb640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 0x7fdd90071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fdda408f540 con 0x7fddac160370 2024-04-25T11:36:05.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.382+0000 7fddb1055640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 0x7fdd90071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:05.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.386+0000 7fddb1055640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 0x7fdd90071910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdda80b2000 tx=0x7fdda80aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:05.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.390+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdda405bd90 con 0x7fddac160370 2024-04-25T11:36:05.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddac06a350 con 0x7fddac160370 2024-04-25T11:36:05.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fdda1ffb640 1 -- 172.21.15.136:0/1561007088 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6100 (secure 0 0 0) 0x7fdda405b690 con 0x7fddac160370 2024-04-25T11:36:05.623 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:05.623 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:03.976061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:resolve","state_seq":33,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:36:05.624 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 63 2024-04-25T11:36:05.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 msgr2=0x7fdd90071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 0x7fdd90071910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdda80b2000 tx=0x7fdda80aa070 comp rx=0 tx=0).stop 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 msgr2=0x7fddac144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdda400ba10 tx=0x7fdda400bed0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 shutdown_connections 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddac160370 0x7fddac144ec0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.626 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fddac15ff60 0x7fddac14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd9006f400 0x7fdd90071910 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 --2- 172.21.15.136:0/1561007088 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fddac15c0d0 0x7fddac1f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:05.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 >> 172.21.15.136:0/1561007088 conn(0x7fddac06f9d0 msgr2=0x7fddac0c7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:05.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 shutdown_connections 2024-04-25T11:36:05.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:05.638+0000 7fddb32b9640 1 -- 172.21.15.136:0/1561007088 wait complete. 2024-04-25T11:36:05.646 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:07.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:07.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 Processor -- start 2024-04-25T11:36:07.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 -- start start 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08f80ca290 0x7f08f80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08f815d750 con 0x7f08f80b5060 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f08f815da30 con 0x7f08f80ca290 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.830+0000 7f08fe5e3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08f815dd10 con 0x7f08f815a430 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08f77fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08f80ca290 0x7f08f80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:51372/0 (socket says 172.21.15.136:51372) 2024-04-25T11:36:07.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 -- 172.21.15.136:0/3615384510 learned_addr learned my addr 172.21.15.136:0/3615384510 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:07.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 -- 172.21.15.136:0/3615384510 >> v2:172.21.15.157:3300/0 conn(0x7f08f80ca290 msgr2=0x7f08f80ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:07.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 --2- 172.21.15.136:0/3615384510 >> v2:172.21.15.157:3300/0 conn(0x7f08f80ca290 0x7f08f80ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 -- 172.21.15.136:0/3615384510 >> v1:172.21.15.157:6789/0 conn(0x7f08f80b5060 legacy=0x7f08f80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 -- 172.21.15.136:0/3615384510 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f815dff0 con 0x7f08f815a430 2024-04-25T11:36:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08fcb80640 1 --2- 172.21.15.136:0/3615384510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009e50 tx=0x7f08f001f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54cdcea9543fd715 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08f6ffd640 1 -- 172.21.15.136:0/3615384510 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f00b4070 con 0x7f08f815a430 2024-04-25T11:36:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08f6ffd640 1 -- 172.21.15.136:0/3615384510 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f00af400 con 0x7f08f815a430 2024-04-25T11:36:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.834+0000 7f08f6ffd640 1 -- 172.21.15.136:0/3615384510 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f00af7d0 con 0x7f08f815a430 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3615384510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 msgr2=0x7f08f815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 --2- 172.21.15.136:0/3615384510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009e50 tx=0x7f08f001f210 comp rx=0 tx=0).stop 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3615384510 shutdown_connections 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 --2- 172.21.15.136:0/3615384510 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f815a840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 --2- 172.21.15.136:0/3615384510 >> v2:172.21.15.157:3300/0 conn(0x7f08f80ca290 0x7f08f80ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3615384510 >> 172.21.15.136:0/3615384510 conn(0x7f08f8070f70 msgr2=0x7f08f8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:07.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3615384510 shutdown_connections 2024-04-25T11:36:07.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3615384510 wait complete. 2024-04-25T11:36:07.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 Processor -- start 2024-04-25T11:36:07.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.838+0000 7f08fe5e3640 1 -- start start 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08f80b5060 0x7f08f81f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f81460d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f08f815ad80 con 0x7f08f80ca290 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08f8159f10 con 0x7f08f80b5060 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08f815d3f0 con 0x7f08f815a430 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51214/0 (socket says 172.21.15.136:51214) 2024-04-25T11:36:07.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 -- 172.21.15.136:0/3109120770 learned_addr learned my addr 172.21.15.136:0/3109120770 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f7fff640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08f80b5060 0x7f08f81f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fcb80640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f81460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 -- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 msgr2=0x7f08f81460d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f81460d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 -- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08f80b5060 msgr2=0x7f08f81f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08f80b5060 0x7f08f81f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 -- 172.21.15.136:0/3109120770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f81f5c40 con 0x7f08f80ca290 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fcb80640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f81460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f77fe640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00e870 tx=0x7f08ec00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec017070 con 0x7f08f80ca290 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ec00c940 con 0x7f08f80ca290 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3109120770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f81f5f20 con 0x7f08f80ca290 2024-04-25T11:36:07.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec00ccf0 con 0x7f08f80ca290 2024-04-25T11:36:07.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08fe5e3640 1 -- 172.21.15.136:0/3109120770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f814a560 con 0x7f08f80ca290 2024-04-25T11:36:07.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d00052a0 con 0x7f08f80ca290 2024-04-25T11:36:07.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.842+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f08ec01a070 con 0x7f08f80ca290 2024-04-25T11:36:07.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.846+0000 7f08f4ff9640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 0x7f08e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:07.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.846+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f08ec08faf0 con 0x7f08f80ca290 2024-04-25T11:36:07.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.846+0000 7f08f7fff640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 0x7f08e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:07.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.846+0000 7f08f7fff640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 0x7f08e4071910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007fd0 tx=0x7f08e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:07.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:07.850+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08ec05bd70 con 0x7f08f80ca290 2024-04-25T11:36:08.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.074+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08d0005860 con 0x7f08f80ca290 2024-04-25T11:36:08.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08f4ff9640 1 -- 172.21.15.136:0/3109120770 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6100 (secure 0 0 0) 0x7f08ec060070 con 0x7f08f80ca290 2024-04-25T11:36:08.062 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:08.062 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:03.976061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:resolve","state_seq":33,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:36:08.063 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 63 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 msgr2=0x7f08e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 0x7f08e4071910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007fd0 tx=0x7f08e8009040 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 msgr2=0x7f08f8148280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00e870 tx=0x7f08ec00ec70 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 shutdown_connections 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08e406f400 0x7f08e4071910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08f815a430 0x7f08f81460d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80ca290 0x7f08f8148280 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 --2- 172.21.15.136:0/3109120770 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08f80b5060 0x7f08f81f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 >> 172.21.15.136:0/3109120770 conn(0x7f08f8070f70 msgr2=0x7f08f80c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 shutdown_connections 2024-04-25T11:36:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:08.078+0000 7f08de7fc640 1 -- 172.21.15.136:0/3109120770 wait complete. 2024-04-25T11:36:08.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:10.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:10.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 Processor -- start 2024-04-25T11:36:10.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 -- start start 2024-04-25T11:36:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2fe40ce260 0x7f2fe40ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40ca010 0x7f2fe40ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe40cbad0 con 0x7f2fe40c8cd0 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2fe40c8550 con 0x7f2fe40ce260 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fec4c8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe40c7c20 con 0x7f2fe40ca010 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2fe40ce260 0x7f2fe40ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2fe40ce260 0x7f2fe40ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51216/0 (socket says 172.21.15.136:51216) 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 -- 172.21.15.136:0/4138269926 learned_addr learned my addr 172.21.15.136:0/4138269926 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2feaa65640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40ca010 0x7f2fe40ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 -- 172.21.15.136:0/4138269926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40ca010 msgr2=0x7f2fe40ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40ca010 0x7f2fe40ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 -- 172.21.15.136:0/4138269926 >> v1:172.21.15.157:6789/0 conn(0x7f2fe40c8cd0 legacy=0x7f2fe414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.266+0000 7f2fe9a63640 1 -- 172.21.15.136:0/4138269926 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe40c7f00 con 0x7f2fe40ce260 2024-04-25T11:36:10.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fe9a63640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ce260 0x7f2fe40ce670 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c86d0 tx=0x7f2fd40213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa8946738ac9be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:10.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fe9262640 1 -- 172.21.15.136:0/4138269926 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40b6070 con 0x7f2fe40ce260 2024-04-25T11:36:10.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fe9262640 1 -- 172.21.15.136:0/4138269926 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd40b1600 con 0x7f2fe40ce260 2024-04-25T11:36:10.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fe9262640 1 -- 172.21.15.136:0/4138269926 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40b19d0 con 0x7f2fe40ce260 2024-04-25T11:36:10.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 -- 172.21.15.136:0/4138269926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ce260 msgr2=0x7f2fe40ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:10.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ce260 0x7f2fe40ce670 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40c86d0 tx=0x7f2fd40213c0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 -- 172.21.15.136:0/4138269926 shutdown_connections 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40ca010 0x7f2fe40ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/4138269926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ce260 0x7f2fe40ce670 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 -- 172.21.15.136:0/4138269926 >> 172.21.15.136:0/4138269926 conn(0x7f2fe406f9d0 msgr2=0x7f2fe406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 -- 172.21.15.136:0/4138269926 shutdown_connections 2024-04-25T11:36:10.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.270+0000 7f2fec4c8640 1 -- 172.21.15.136:0/4138269926 wait complete. 2024-04-25T11:36:10.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 Processor -- start 2024-04-25T11:36:10.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- start start 2024-04-25T11:36:10.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 0x7f2fe4155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 0x7f2fe415b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe40cb720 con 0x7f2fe40ca010 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe40ca960 con 0x7f2fe40ce260 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe414d990 con 0x7f2fe40c8cd0 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51230/0 (socket says 172.21.15.136:51230) 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 -- 172.21.15.136:0/3073406609 learned_addr learned my addr 172.21.15.136:0/3073406609 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fea264640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 0x7f2fe4155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2feaa65640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 0x7f2fe415b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 -- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 msgr2=0x7f2fe4155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 0x7f2fe4155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 -- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 msgr2=0x7f2fe415b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 0x7f2fe415b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 -- 172.21.15.136:0/3073406609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe414cdb0 con 0x7f2fe40ca010 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fea264640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 0x7f2fe4155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:10.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2feaa65640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 0x7f2fe415b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fe9a63640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400b2c0 tx=0x7f2fd401e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40b6040 con 0x7f2fe40ca010 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd40c2070 con 0x7f2fe40ca010 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe4156440 con 0x7f2fe40ca010 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40ba840 con 0x7f2fe40ca010 2024-04-25T11:36:10.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.274+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe414d050 con 0x7f2fe40ca010 2024-04-25T11:36:10.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fcc0052a0 con 0x7f2fe40ca010 2024-04-25T11:36:10.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2fd40b9070 con 0x7f2fe40ca010 2024-04-25T11:36:10.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fdaffd640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 0x7f2fc8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:10.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f2fd412ccf0 con 0x7f2fe40ca010 2024-04-25T11:36:10.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fea264640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 0x7f2fc8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:10.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.278+0000 7f2fea264640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 0x7f2fc8071910 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098b0 tx=0x7f2fe0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:10.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.282+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2fd4131070 con 0x7f2fe40ca010 2024-04-25T11:36:10.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.530+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fcc005860 con 0x7f2fe40ca010 2024-04-25T11:36:10.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.530+0000 7f2fdaffd640 1 -- 172.21.15.136:0/3073406609 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6102 (secure 0 0 0) 0x7f2fd401af20 con 0x7f2fe40ca010 2024-04-25T11:36:10.515 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:10.516 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:09.267444+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:10.516 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 64 2024-04-25T11:36:10.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 msgr2=0x7f2fc8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:10.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 0x7f2fc8071910 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098b0 tx=0x7f2fe0008040 comp rx=0 tx=0).stop 2024-04-25T11:36:10.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 msgr2=0x7f2fe41538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400b2c0 tx=0x7f2fd401e160 comp rx=0 tx=0).stop 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 shutdown_connections 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fe40ce260 0x7f2fe415b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fe40ca010 0x7f2fe41538f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2fc806f400 0x7f2fc8071910 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 --2- 172.21.15.136:0/3073406609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fe40c8cd0 0x7f2fe4155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 >> 172.21.15.136:0/3073406609 conn(0x7f2fe406f9d0 msgr2=0x7f2fe414bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 shutdown_connections 2024-04-25T11:36:10.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:10.534+0000 7f2fec4c8640 1 -- 172.21.15.136:0/3073406609 wait complete. 2024-04-25T11:36:10.538 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:12.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.718+0000 7f6ad000e640 1 Processor -- start 2024-04-25T11:36:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.718+0000 7f6ad000e640 1 -- start start 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ad000e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 0x7f6ac80c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ad000e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac8164700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ad000e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac80c8d40 con 0x7f6ac815a5f0 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ad000e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac80c8fd0 con 0x7f6ac8154060 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ad000e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac80c3880 con 0x7f6ac815aa00 2024-04-25T11:36:12.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6acddaa640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 0x7f6ac80c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ace5ab640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac8164700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6acddaa640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 0x7f6ac80c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:54470/0 (socket says 172.21.15.136:54470) 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6acddaa640 1 -- 172.21.15.136:0/454414385 learned_addr learned my addr 172.21.15.136:0/454414385 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ace5ab640 1 -- 172.21.15.136:0/454414385 >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 msgr2=0x7f6ac80c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ace5ab640 1 --2- 172.21.15.136:0/454414385 >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 0x7f6ac80c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ace5ab640 1 -- 172.21.15.136:0/454414385 >> v1:172.21.15.157:6789/0 conn(0x7f6ac815a5f0 legacy=0x7f6ac80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.722+0000 7f6ace5ab640 1 -- 172.21.15.136:0/454414385 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac806d9b0 con 0x7f6ac815aa00 2024-04-25T11:36:12.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ace5ab640 1 --2- 172.21.15.136:0/454414385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac8164700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009d90 tx=0x7f6ac001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54225c5b6732e1d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:12.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6accda8640 1 -- 172.21.15.136:0/454414385 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00b4070 con 0x7f6ac815aa00 2024-04-25T11:36:12.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6accda8640 1 -- 172.21.15.136:0/454414385 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac00af600 con 0x7f6ac815aa00 2024-04-25T11:36:12.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6accda8640 1 -- 172.21.15.136:0/454414385 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00af9d0 con 0x7f6ac815aa00 2024-04-25T11:36:12.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 -- 172.21.15.136:0/454414385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 msgr2=0x7f6ac8164700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 --2- 172.21.15.136:0/454414385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac8164700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009d90 tx=0x7f6ac001f150 comp rx=0 tx=0).stop 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 -- 172.21.15.136:0/454414385 shutdown_connections 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 --2- 172.21.15.136:0/454414385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac8164700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 --2- 172.21.15.136:0/454414385 >> v2:172.21.15.157:3300/0 conn(0x7f6ac8154060 0x7f6ac80c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 -- 172.21.15.136:0/454414385 >> 172.21.15.136:0/454414385 conn(0x7f6ac80ca5a0 msgr2=0x7f6ac80ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 -- 172.21.15.136:0/454414385 shutdown_connections 2024-04-25T11:36:12.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.726+0000 7f6ad000e640 1 -- 172.21.15.136:0/454414385 wait complete. 2024-04-25T11:36:12.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 Processor -- start 2024-04-25T11:36:12.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 -- start start 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ac8154060 0x7f6ac81f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac80ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac806df30 con 0x7f6ac815a5f0 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac8154470 con 0x7f6ac8154060 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac815a0d0 con 0x7f6ac815aa00 2024-04-25T11:36:12.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51240/0 (socket says 172.21.15.136:51240) 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 -- 172.21.15.136:0/4220049520 learned_addr learned my addr 172.21.15.136:0/4220049520 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ace5ab640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac80ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acddaa640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ac8154060 0x7f6ac81f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 -- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 msgr2=0x7f6ac80ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac80ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 -- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ac8154060 msgr2=0x7f6ac81f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ac8154060 0x7f6ac81f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 -- 172.21.15.136:0/4220049520 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac81f6640 con 0x7f6ac815a5f0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6acd5a9640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c340 tx=0x7f6ab800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80052d0 con 0x7f6ac815a5f0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab8005470 con 0x7f6ac815a5f0 2024-04-25T11:36:12.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac81548a0 con 0x7f6ac815a5f0 2024-04-25T11:36:12.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.730+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8007260 con 0x7f6ac815a5f0 2024-04-25T11:36:12.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac8154e60 con 0x7f6ac815a5f0 2024-04-25T11:36:12.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac806a350 con 0x7f6ac815a5f0 2024-04-25T11:36:12.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6ab8007400 con 0x7f6ac815a5f0 2024-04-25T11:36:12.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6abe7fc640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 0x7f6aac071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:12.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f6ab808eeb0 con 0x7f6ac815a5f0 2024-04-25T11:36:12.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6acddaa640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 0x7f6aac071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:12.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.734+0000 7f6acddaa640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 0x7f6aac071910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4009a40 tx=0x7f6ac4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:12.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.738+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6ab805b670 con 0x7f6ac815a5f0 2024-04-25T11:36:12.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ac80c85e0 con 0x7f6ac815a5f0 2024-04-25T11:36:12.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6abe7fc640 1 -- 172.21.15.136:0/4220049520 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6102 (secure 0 0 0) 0x7f6ab805af70 con 0x7f6ac815a5f0 2024-04-25T11:36:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:12.983 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:09.267444+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:12.983 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 64 2024-04-25T11:36:12.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 msgr2=0x7f6aac071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.985 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 0x7f6aac071910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4009a40 tx=0x7f6ac4008040 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 msgr2=0x7f6ac80c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c340 tx=0x7f6ab800c800 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 shutdown_connections 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6aac06f400 0x7f6aac071910 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ac815aa00 0x7f6ac80ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ac815a5f0 0x7f6ac80c4fa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 --2- 172.21.15.136:0/4220049520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ac8154060 0x7f6ac81f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 >> 172.21.15.136:0/4220049520 conn(0x7f6ac80ca5a0 msgr2=0x7f6ac8158d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 shutdown_connections 2024-04-25T11:36:12.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:12.998+0000 7f6ad000e640 1 -- 172.21.15.136:0/4220049520 wait complete. 2024-04-25T11:36:13.005 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5249": { "addr": "172.21.15.157:6837/3576995854", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3576995854, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3576995854, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5249, "incarnation": 62, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:reconnect", "state_seq": 35 }, "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:36:09.267444+0000", "qdb_cluster": [ 5356, 5385, 5449, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 5249 } } } ], "standbys": [] } 2024-04-25T11:36:15.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:15.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 Processor -- start 2024-04-25T11:36:15.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 -- start start 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6480ca290 0x7fd6480ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd64815a430 0x7fd64815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd64815d750 con 0x7fd6480b5060 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd64815da30 con 0x7fd64815a430 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd65043a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd64815dd10 con 0x7fd6480ca290 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd64815a430 0x7fd64815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd64815a430 0x7fd64815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35472/0 (socket says 172.21.15.136:35472) 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 -- 172.21.15.136:0/178112696 learned_addr learned my addr 172.21.15.136:0/178112696 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64d9d5640 1 --2- 172.21.15.136:0/178112696 >> v2:172.21.15.157:3300/0 conn(0x7fd6480ca290 0x7fd6480ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 -- 172.21.15.136:0/178112696 >> v2:172.21.15.157:3300/0 conn(0x7fd6480ca290 msgr2=0x7fd6480ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 --2- 172.21.15.136:0/178112696 >> v2:172.21.15.157:3300/0 conn(0x7fd6480ca290 0x7fd6480ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 -- 172.21.15.136:0/178112696 >> v1:172.21.15.157:6789/0 conn(0x7fd6480b5060 legacy=0x7fd6480c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.198+0000 7fd64e9d7640 1 -- 172.21.15.136:0/178112696 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64815dff0 con 0x7fd64815a430 2024-04-25T11:36:15.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd64e9d7640 1 --2- 172.21.15.136:0/178112696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd64815a430 0x7fd64815a840 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd6480d00d0 tx=0x7fd64001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789cb38e8d2f96a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd64d1d4640 1 -- 172.21.15.136:0/178112696 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b4070 con 0x7fd64815a430 2024-04-25T11:36:15.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd64d1d4640 1 -- 172.21.15.136:0/178112696 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6400af400 con 0x7fd64815a430 2024-04-25T11:36:15.188 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd64d1d4640 1 -- 172.21.15.136:0/178112696 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6400af7d0 con 0x7fd64815a430 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 -- 172.21.15.136:0/178112696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd64815a430 msgr2=0x7fd64815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 --2- 172.21.15.136:0/178112696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd64815a430 0x7fd64815a840 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd6480d00d0 tx=0x7fd64001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 -- 172.21.15.136:0/178112696 shutdown_connections 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 --2- 172.21.15.136:0/178112696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd64815a430 0x7fd64815a840 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 --2- 172.21.15.136:0/178112696 >> v2:172.21.15.157:3300/0 conn(0x7fd6480ca290 0x7fd6480ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 -- 172.21.15.136:0/178112696 >> 172.21.15.136:0/178112696 conn(0x7fd648070f70 msgr2=0x7fd648068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 -- 172.21.15.136:0/178112696 shutdown_connections 2024-04-25T11:36:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.202+0000 7fd65043a640 1 -- 172.21.15.136:0/178112696 wait complete. 2024-04-25T11:36:15.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 Processor -- start 2024-04-25T11:36:15.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 -- start start 2024-04-25T11:36:15.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6480b5060 0x7fd64814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64815a430 0x7fd6481480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd64815ad80 con 0x7fd6480ca290 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd648159f10 con 0x7fd6480b5060 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd64815d3f0 con 0x7fd64815a430 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35482/0 (socket says 172.21.15.136:35482) 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 -- 172.21.15.136:0/531841746 learned_addr learned my addr 172.21.15.136:0/531841746 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64e1d6640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6480b5060 0x7fd64814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64e9d7640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64815a430 0x7fd6481480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 -- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64815a430 msgr2=0x7fd6481480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64815a430 0x7fd6481480e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 -- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6480b5060 msgr2=0x7fd64814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6480b5060 0x7fd64814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 -- 172.21.15.136:0/531841746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd648151bd0 con 0x7fd6480ca290 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd64d9d5640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d620 tx=0x7fd63800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63800b620 con 0x7fd6480ca290 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64815c120 con 0x7fd6480ca290 2024-04-25T11:36:15.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.206+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638019070 con 0x7fd6480ca290 2024-04-25T11:36:15.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63800b9e0 con 0x7fd6480ca290 2024-04-25T11:36:15.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6481f4800 con 0x7fd6480ca290 2024-04-25T11:36:15.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd648151ef0 con 0x7fd6480ca290 2024-04-25T11:36:15.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd63800bb80 con 0x7fd6480ca290 2024-04-25T11:36:15.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd63effd640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 0x7fd62c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:15.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fd63808fdc0 con 0x7fd6480ca290 2024-04-25T11:36:15.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.210+0000 7fd64e1d6640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 0x7fd62c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:15.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.214+0000 7fd64e1d6640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 0x7fd62c071910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd6440015b0 tx=0x7fd644009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:15.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.218+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd63805c610 con 0x7fd6480ca290 2024-04-25T11:36:15.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd64806a350 con 0x7fd6480ca290 2024-04-25T11:36:15.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd63effd640 1 -- 172.21.15.136:0/531841746 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6099 (secure 0 0 0) 0x7fd63805bf10 con 0x7fd6480ca290 2024-04-25T11:36:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:15.431 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:14.288019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:36:15.431 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 65 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 msgr2=0x7fd62c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 0x7fd62c071910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd6440015b0 tx=0x7fd644009040 comp rx=0 tx=0).stop 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 msgr2=0x7fd648147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d620 tx=0x7fd63800dae0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 shutdown_connections 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd62c06f400 0x7fd62c071910 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd64815a430 0x7fd6481480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6480ca290 0x7fd648147ba0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 --2- 172.21.15.136:0/531841746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6480b5060 0x7fd64814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:15.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 >> 172.21.15.136:0/531841746 conn(0x7fd648070f70 msgr2=0x7fd6480ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:15.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 shutdown_connections 2024-04-25T11:36:15.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:15.446+0000 7fd65043a640 1 -- 172.21.15.136:0/531841746 wait complete. 2024-04-25T11:36:15.452 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:17.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:17.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.630+0000 7f01a17f3640 1 Processor -- start 2024-04-25T11:36:17.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.630+0000 7f01a17f3640 1 -- start start 2024-04-25T11:36:17.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c0c96e0 0x7f019c1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 0x7f019c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f019c15a3d0 0x7f019c15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f019c06cc10 con 0x7f019c15a3d0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f019c0c9af0 con 0x7f019c0c79e0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f01a17f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f019c0cb220 con 0x7f019c0c96e0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f019c15a3d0 0x7f019c15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f019c15a3d0 0x7f019c15a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35496/0 (socket says 172.21.15.136:35496) 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 -- 172.21.15.136:0/1644159508 learned_addr learned my addr 172.21.15.136:0/1644159508 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019affd640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c0c96e0 0x7f019c1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019a7fc640 1 --2- 172.21.15.136:0/1644159508 >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 0x7f019c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 -- 172.21.15.136:0/1644159508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c0c96e0 msgr2=0x7f019c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c0c96e0 0x7f019c1540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 -- 172.21.15.136:0/1644159508 >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 msgr2=0x7f019c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:17.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 --2- 172.21.15.136:0/1644159508 >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 0x7f019c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 -- 172.21.15.136:0/1644159508 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c0cb500 con 0x7f019c15a3d0 2024-04-25T11:36:17.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019a7fc640 1 --2- 172.21.15.136:0/1644159508 >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 0x7f019c0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:17.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.634+0000 7f019b7fe640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c15a3d0 0x7f019c15a7e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f019c0706f0 tx=0x7f019401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccef6161bc1b9be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f0199ffb640 1 -- 172.21.15.136:0/1644159508 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940b4070 con 0x7f019c15a3d0 2024-04-25T11:36:17.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f0199ffb640 1 -- 172.21.15.136:0/1644159508 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01940af400 con 0x7f019c15a3d0 2024-04-25T11:36:17.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f0199ffb640 1 -- 172.21.15.136:0/1644159508 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940af7d0 con 0x7f019c15a3d0 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- 172.21.15.136:0/1644159508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c15a3d0 msgr2=0x7f019c15a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c15a3d0 0x7f019c15a7e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f019c0706f0 tx=0x7f019401f1e0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- 172.21.15.136:0/1644159508 shutdown_connections 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c15a3d0 0x7f019c15a7e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 --2- 172.21.15.136:0/1644159508 >> v2:172.21.15.157:3300/0 conn(0x7f019c0c79e0 0x7f019c0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 --2- 172.21.15.136:0/1644159508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c0c96e0 0x7f019c1540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- 172.21.15.136:0/1644159508 >> 172.21.15.136:0/1644159508 conn(0x7f019c06f9d0 msgr2=0x7f019c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- 172.21.15.136:0/1644159508 shutdown_connections 2024-04-25T11:36:17.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- 172.21.15.136:0/1644159508 wait complete. 2024-04-25T11:36:17.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 Processor -- start 2024-04-25T11:36:17.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.638+0000 7f01a17f3640 1 -- start start 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f019c0c79e0 0x7f019c0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c15a3d0 0x7f019c154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f019c154610 con 0x7f019c0c96e0 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f019c15ad20 con 0x7f019c0c79e0 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f019c0cf0f0 con 0x7f019c15a3d0 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35508/0 (socket says 172.21.15.136:35508) 2024-04-25T11:36:17.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 -- 172.21.15.136:0/737851953 learned_addr learned my addr 172.21.15.136:0/737851953 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 -- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c15a3d0 msgr2=0x7f019c154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c15a3d0 0x7f019c154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 -- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f019c0c79e0 msgr2=0x7f019c0cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f019c0c79e0 0x7f019c0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 -- 172.21.15.136:0/737851953 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c1f4570 con 0x7f019c0c96e0 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f019a7fc640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f019000c820 tx=0x7f019000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190015070 con 0x7f019c0c96e0 2024-04-25T11:36:17.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0190007a30 con 0x7f019c0c96e0 2024-04-25T11:36:17.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019c155290 con 0x7f019c0c96e0 2024-04-25T11:36:17.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190007e20 con 0x7f019c0c96e0 2024-04-25T11:36:17.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.642+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019c0cdba0 con 0x7f019c0c96e0 2024-04-25T11:36:17.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019c06a350 con 0x7f019c0c96e0 2024-04-25T11:36:17.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0190018070 con 0x7f019c0c96e0 2024-04-25T11:36:17.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f01837fe640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 0x7f0188071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:17.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f019008fc00 con 0x7f019c0c96e0 2024-04-25T11:36:17.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f019affd640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 0x7f0188071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:17.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.646+0000 7f019affd640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 0x7f0188071910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f018c00b850 tx=0x7f018c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:17.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.654+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f019c1f4570 con 0x7f019c0c96e0 2024-04-25T11:36:17.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.878+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f019c158b20 con 0x7f019c0c96e0 2024-04-25T11:36:17.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.878+0000 7f01837fe640 1 -- 172.21.15.136:0/737851953 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6099 (secure 0 0 0) 0x7f019005c520 con 0x7f019c0c96e0 2024-04-25T11:36:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:17.865 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:14.288019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:36:17.865 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 65 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 msgr2=0x7f0188071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 0x7f0188071910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f018c00b850 tx=0x7f018c008040 comp rx=0 tx=0).stop 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 msgr2=0x7f019c1f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f019000c820 tx=0x7f019000cce0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 shutdown_connections 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f018806f400 0x7f0188071910 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f019c15a3d0 0x7f019c154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f019c0c96e0 0x7f019c1f4030 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 --2- 172.21.15.136:0/737851953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f019c0c79e0 0x7f019c0cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:17.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 >> 172.21.15.136:0/737851953 conn(0x7f019c06f9d0 msgr2=0x7f019c159ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:17.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 shutdown_connections 2024-04-25T11:36:17.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:17.882+0000 7f01a17f3640 1 -- 172.21.15.136:0/737851953 wait complete. 2024-04-25T11:36:17.884 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:36:19.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:20.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.065+0000 7f471d933640 1 Processor -- start 2024-04-25T11:36:20.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.065+0000 7f471d933640 1 -- start start 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f471d933640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 0x7f47180c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f471d933640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f471815a5f0 0x7f47180c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f471d933640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47180c8d40 con 0x7f471815aa00 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f471d933640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f47180c8fd0 con 0x7f471815a5f0 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f471d933640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f47180c3880 con 0x7f4718154060 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f471815a5f0 0x7f47180c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47177fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f471815aa00 0x7f471815d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:58304/0 (socket says 172.21.15.136:58304) 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47177fe640 1 -- 172.21.15.136:0/3779809271 learned_addr learned my addr 172.21.15.136:0/3779809271 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4716ffd640 1 --2- 172.21.15.136:0/3779809271 >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 0x7f47180c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 -- 172.21.15.136:0/3779809271 >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 msgr2=0x7f47180c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:20.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 --2- 172.21.15.136:0/3779809271 >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 0x7f47180c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 -- 172.21.15.136:0/3779809271 >> v1:172.21.15.136:6789/0 conn(0x7f471815aa00 legacy=0x7f471815d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 -- 172.21.15.136:0/3779809271 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471806d9b0 con 0x7f471815a5f0 2024-04-25T11:36:20.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4715ffb640 1 -- 172.21.15.136:0/3779809271 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182991997 0 0) 0x7f47180c8d40 con 0x7f471815aa00 2024-04-25T11:36:20.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4716ffd640 1 --2- 172.21.15.136:0/3779809271 >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 0x7f47180c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:20.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f47167fc640 1 --2- 172.21.15.136:0/3779809271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f471815a5f0 0x7f47180c71b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f47180c7c60 tx=0x7f470c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce09f730a86b1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4715ffb640 1 -- 172.21.15.136:0/3779809271 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c0b5070 con 0x7f471815a5f0 2024-04-25T11:36:20.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4715ffb640 1 -- 172.21.15.136:0/3779809271 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470c0a8970 con 0x7f471815a5f0 2024-04-25T11:36:20.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.069+0000 7f4715ffb640 1 -- 172.21.15.136:0/3779809271 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c0a8d40 con 0x7f471815a5f0 2024-04-25T11:36:20.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- 172.21.15.136:0/3779809271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f471815a5f0 msgr2=0x7f47180c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 --2- 172.21.15.136:0/3779809271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f471815a5f0 0x7f47180c71b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f47180c7c60 tx=0x7f470c01ec70 comp rx=0 tx=0).stop 2024-04-25T11:36:20.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- 172.21.15.136:0/3779809271 shutdown_connections 2024-04-25T11:36:20.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 --2- 172.21.15.136:0/3779809271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f471815a5f0 0x7f47180c71b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 --2- 172.21.15.136:0/3779809271 >> v2:172.21.15.157:3300/0 conn(0x7f4718154060 0x7f47180c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- 172.21.15.136:0/3779809271 >> 172.21.15.136:0/3779809271 conn(0x7f47180ca5a0 msgr2=0x7f47180ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- 172.21.15.136:0/3779809271 shutdown_connections 2024-04-25T11:36:20.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- 172.21.15.136:0/3779809271 wait complete. 2024-04-25T11:36:20.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 Processor -- start 2024-04-25T11:36:20.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.073+0000 7f471d933640 1 -- start start 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f471815a5f0 0x7f47181e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471815aa00 0x7f47181e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f471806df30 con 0x7f4718154060 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4718154470 con 0x7f471815a5f0 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f471815a0d0 con 0x7f471815aa00 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35524/0 (socket says 172.21.15.136:35524) 2024-04-25T11:36:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 -- 172.21.15.136:0/126641079 learned_addr learned my addr 172.21.15.136:0/126641079 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f47167fc640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f471815a5f0 0x7f47181e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f47177fe640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471815aa00 0x7f47181e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 -- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471815aa00 msgr2=0x7f47181e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471815aa00 0x7f47181e3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 -- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f471815a5f0 msgr2=0x7f47181e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f471815a5f0 0x7f47181e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 -- 172.21.15.136:0/126641079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47181f14f0 con 0x7f4718154060 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f4716ffd640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f470800d800 tx=0x7f470800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708018020 con 0x7f4718154060 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47080096c0 con 0x7f4718154060 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47181f07d0 con 0x7f4718154060 2024-04-25T11:36:20.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708009ab0 con 0x7f4718154060 2024-04-25T11:36:20.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47181e3230 con 0x7f4718154060 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f471806a350 con 0x7f4718154060 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f470801b020 con 0x7f4718154060 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.077+0000 7f46fffff640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 0x7f4704071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.081+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f470808fdd0 con 0x7f4718154060 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.081+0000 7f47167fc640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 0x7f4704071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:20.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.081+0000 7f47167fc640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 0x7f4704071910 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f470c0b2000 tx=0x7f470c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:20.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.085+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4708022020 con 0x7f4718154060 2024-04-25T11:36:20.174 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:36:20.188+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:36:20.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.309+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47181486b0 con 0x7f4718154060 2024-04-25T11:36:20.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.309+0000 7f46fffff640 1 -- 172.21.15.136:0/126641079 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6104 (secure 0 0 0) 0x7f470805c730 con 0x7f4718154060 2024-04-25T11:36:20.297 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:20.297 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5249,5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:20.297 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 66 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 msgr2=0x7f4704071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 0x7f4704071910 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f470c0b2000 tx=0x7f470c008040 comp rx=0 tx=0).stop 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 msgr2=0x7f47181f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f470800d800 tx=0x7f470800dcc0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 shutdown_connections 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f470406f400 0x7f4704071910 secure :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f470c0b2000 tx=0x7f470c008040 comp rx=0 tx=0).stop 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f471815aa00 0x7f47181e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f471815a5f0 0x7f47181e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 --2- 172.21.15.136:0/126641079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4718154060 0x7f47181f0fb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:20.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 >> 172.21.15.136:0/126641079 conn(0x7f47180ca5a0 msgr2=0x7f4718157a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:20.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 shutdown_connections 2024-04-25T11:36:20.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:20.313+0000 7f471d933640 1 -- 172.21.15.136:0/126641079 wait complete. 2024-04-25T11:36:20.326 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=5238 2024-04-25T11:36:20.326 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.c 2024-04-25T11:36:22.327 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-25T11:36:22.327 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T11:36:22.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T11:36:22.330 INFO:tasks.ceph.mds.c:Started 2024-04-25T11:36:22.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:22.447 INFO:tasks.ceph.mds.c.smithi136.stdout:starting mds.c at 2024-04-25T11:36:22.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 Processor -- start 2024-04-25T11:36:22.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 -- start start 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcda40ca010 0x7fcda406cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 0x7fcda406d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda40c8a20 con 0x7fcda415bfe0 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fcda406c670 con 0x7fcda40ca010 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda998b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcda40c8d00 con 0x7fcda40c9b50 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda2ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcda40ca010 0x7fcda406cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fcda415bfe0 0x7fcda415c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:58318/0 (socket says 172.21.15.136:58318) 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/2597884226 >> v2:172.21.15.136:3300/0 conn(0x7fcda40ca010 0x7fcda406cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35544/0 (socket says 172.21.15.136:35544) 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda37fe640 1 -- 172.21.15.136:0/2597884226 learned_addr learned my addr 172.21.15.136:0/2597884226 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda27fc640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 0x7fcda406d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda2ffd640 1 -- 172.21.15.136:0/2597884226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 msgr2=0x7fcda406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 0x7fcda406d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda2ffd640 1 -- 172.21.15.136:0/2597884226 >> v1:172.21.15.136:6789/0 conn(0x7fcda415bfe0 legacy=0x7fcda415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda2ffd640 1 -- 172.21.15.136:0/2597884226 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda4074d10 con 0x7fcda40ca010 2024-04-25T11:36:22.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.509+0000 7fcda1ffb640 1 -- 172.21.15.136:0/2597884226 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297820993 0 0) 0x7fcda40c8a20 con 0x7fcda415bfe0 2024-04-25T11:36:22.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda27fc640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 0x7fcda406d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:22.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda406cb00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd940097c0 tx=0x7fcd9401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbda7613b7c678a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:22.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda1ffb640 1 -- 172.21.15.136:0/2597884226 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd940b1070 con 0x7fcda40ca010 2024-04-25T11:36:22.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda1ffb640 1 -- 172.21.15.136:0/2597884226 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd940ac630 con 0x7fcda40ca010 2024-04-25T11:36:22.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda1ffb640 1 -- 172.21.15.136:0/2597884226 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd940a8050 con 0x7fcda40ca010 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 -- 172.21.15.136:0/2597884226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 msgr2=0x7fcda406cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda406cb00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd940097c0 tx=0x7fcd9401c4c0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 -- 172.21.15.136:0/2597884226 shutdown_connections 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40c9b50 0x7fcda406d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 --2- 172.21.15.136:0/2597884226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda406cb00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 -- 172.21.15.136:0/2597884226 >> 172.21.15.136:0/2597884226 conn(0x7fcda40ca5a0 msgr2=0x7fcda40ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 -- 172.21.15.136:0/2597884226 shutdown_connections 2024-04-25T11:36:22.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.513+0000 7fcda998b640 1 -- 172.21.15.136:0/2597884226 wait complete. 2024-04-25T11:36:22.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.517+0000 7fcda998b640 1 Processor -- start 2024-04-25T11:36:22.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.517+0000 7fcda998b640 1 -- start start 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda40c9b50 0x7fcda4153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda415bfe0 0x7fcda4155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcda41694d0 con 0x7fcda40ca010 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcda4169350 con 0x7fcda40c9b50 2024-04-25T11:36:22.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcda4169650 con 0x7fcda415bfe0 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35554/0 (socket says 172.21.15.136:35554) 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 -- 172.21.15.136:0/120412198 learned_addr learned my addr 172.21.15.136:0/120412198 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda40c9b50 0x7fcda4153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda37fe640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda415bfe0 0x7fcda4155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 -- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda415bfe0 msgr2=0x7fcda4155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda415bfe0 0x7fcda4155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 -- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda40c9b50 msgr2=0x7fcda4153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda40c9b50 0x7fcda4153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 -- 172.21.15.136:0/120412198 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda41f8c40 con 0x7fcda40ca010 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda27fc640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800d7d0 tx=0x7fcd9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980096a0 con 0x7fcda40ca010 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd98019070 con 0x7fcda40ca010 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda41f8f20 con 0x7fcda40ca010 2024-04-25T11:36:22.508 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98009a60 con 0x7fcda40ca010 2024-04-25T11:36:22.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda41f94e0 con 0x7fcda40ca010 2024-04-25T11:36:22.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda406a350 con 0x7fcda40ca010 2024-04-25T11:36:22.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcd98009c00 con 0x7fcda40ca010 2024-04-25T11:36:22.510 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.521+0000 7fcd8bfff640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 0x7fcd90071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:22.510 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.525+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fcd9808fc00 con 0x7fcda40ca010 2024-04-25T11:36:22.510 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.525+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 0x7fcd90071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:22.511 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.525+0000 7fcda2ffd640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 0x7fcd90071910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ae000 tx=0x7fcd940a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:22.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.529+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcd98096080 con 0x7fcda40ca010 2024-04-25T11:36:22.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.809+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcda4154420 con 0x7fcda40ca010 2024-04-25T11:36:22.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.809+0000 7fcd8bfff640 1 -- 172.21.15.136:0/120412198 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7fcd9805be80 con 0x7fcda40ca010 2024-04-25T11:36:22.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:22.794 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:22.794 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:22.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 msgr2=0x7fcd90071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 0x7fcd90071910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcd940ae000 tx=0x7fcd940a7000 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 msgr2=0x7fcda41f5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800d7d0 tx=0x7fcd9800dc90 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 shutdown_connections 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fcd9006f400 0x7fcd90071910 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda415bfe0 0x7fcda4155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcda40ca010 0x7fcda41f5550 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 --2- 172.21.15.136:0/120412198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda40c9b50 0x7fcda4153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 >> 172.21.15.136:0/120412198 conn(0x7fcda40ca5a0 msgr2=0x7fcda415af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 shutdown_connections 2024-04-25T11:36:22.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:22.813+0000 7fcda998b640 1 -- 172.21.15.136:0/120412198 wait complete. 2024-04-25T11:36:22.817 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-25T11:36:22.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2024-04-25T11:36:23.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:36:23.928 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:36:46.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:46.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.009+0000 7fde2db59640 1 Processor -- start 2024-04-25T11:36:46.992 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.009+0000 7fde2db59640 1 -- start start 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde2db59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde280c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde2db59640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2815d750 con 0x7fde280ca290 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde2db59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde2815da30 con 0x7fde2815ab40 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde2db59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde2815dd10 con 0x7fde280b5060 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fde280ca290 0x7fde2815a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46700/0 (socket says 172.21.15.136:46700) 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 -- 172.21.15.136:0/4100389162 learned_addr learned my addr 172.21.15.136:0/4100389162 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde277fe640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde280c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:46.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762495411 0 0) 0x7fde2815d750 con 0x7fde280ca290 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde10003530 con 0x7fde280ca290 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3598793313 0 0) 0x7fde10003530 con 0x7fde280ca290 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 msgr2=0x7fde280c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde280c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 >> v1:172.21.15.157:6789/0 conn(0x7fde2815ab40 legacy=0x7fde28164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2815dff0 con 0x7fde280ca290 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fde1c001680 con 0x7fde280ca290 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 >> v1:172.21.15.136:6789/0 conn(0x7fde280ca290 legacy=0x7fde2815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde100039f0 0x7fde10003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde10004810 0x7fde10004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde10005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde2815dd10 con 0x7fde10004810 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde1000bb90 con 0x7fde100039f0 2024-04-25T11:36:46.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde1000bd90 con 0x7fde280b5060 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde10004810 0x7fde10004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde277fe640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde100039f0 0x7fde10003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde27fff640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde10005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 -- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 msgr2=0x7fde10005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280b5060 0x7fde10005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 -- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde100039f0 msgr2=0x7fde10003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde100039f0 0x7fde10003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde26ffd640 1 -- 172.21.15.136:0/4100389162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde10003530 con 0x7fde10004810 2024-04-25T11:36:46.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.013+0000 7fde277fe640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde100039f0 0x7fde10003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:46.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde26ffd640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde10004810 0x7fde10004c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fde1c001ec0 tx=0x7fde1c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7ffa408e1170fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:46.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c007d40 con 0x7fde10004810 2024-04-25T11:36:46.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1c0a9070 con 0x7fde10004810 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde267fc640 1 -- 172.21.15.136:0/4100389162 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0a5dc0 con 0x7fde10004810 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- 172.21.15.136:0/4100389162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde10004810 msgr2=0x7fde10004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 --2- 172.21.15.136:0/4100389162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde10004810 0x7fde10004c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fde1c001ec0 tx=0x7fde1c01cc60 comp rx=0 tx=0).stop 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde27fff640 1 -- 172.21.15.136:0/4100389162 reap_dead start 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- 172.21.15.136:0/4100389162 shutdown_connections 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- 172.21.15.136:0/4100389162 >> 172.21.15.136:0/4100389162 conn(0x7fde28070f70 msgr2=0x7fde28068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- 172.21.15.136:0/4100389162 shutdown_connections 2024-04-25T11:36:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- 172.21.15.136:0/4100389162 wait complete. 2024-04-25T11:36:47.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 Processor -- start 2024-04-25T11:36:47.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.017+0000 7fde2db59640 1 -- start start 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280cef40 0x7fde2814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde28147c70 0x7fde28144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde2815d3f0 con 0x7fde28153780 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde28159f10 con 0x7fde28147c70 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde280ca6a0 con 0x7fde280cef40 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47178/0 (socket says 172.21.15.136:47178) 2024-04-25T11:36:47.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 -- 172.21.15.136:0/48255361 learned_addr learned my addr 172.21.15.136:0/48255361 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde277fe640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280cef40 0x7fde2814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde27fff640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde28147c70 0x7fde28144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 -- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280cef40 msgr2=0x7fde2814ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280cef40 0x7fde2814ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 -- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde28147c70 msgr2=0x7fde28144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde28147c70 0x7fde28144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 -- 172.21.15.136:0/48255361 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde280cd300 con 0x7fde28153780 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde26ffd640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fde1c01cca0 tx=0x7fde10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c0bc070 con 0x7fde28153780 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1c000ee0 con 0x7fde28153780 2024-04-25T11:36:47.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde280c4da0 con 0x7fde28153780 2024-04-25T11:36:47.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c007380 con 0x7fde28153780 2024-04-25T11:36:47.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2815b060 con 0x7fde28153780 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.021+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2806a350 con 0x7fde28153780 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.025+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde1c0bf070 con 0x7fde28153780 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.025+0000 7fde0ffff640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 0x7fde14071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.025+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fde1c129650 con 0x7fde28153780 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.025+0000 7fde277fe640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 0x7fde14071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.025+0000 7fde277fe640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 0x7fde14071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde280b56d0 tx=0x7fde18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.029+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde1c016970 con 0x7fde28153780 2024-04-25T11:36:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.281+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde2814dd20 con 0x7fde28153780 2024-04-25T11:36:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.281+0000 7fde0ffff640 1 -- 172.21.15.136:0/48255361 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7fde1c0f6090 con 0x7fde28153780 2024-04-25T11:36:47.264 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:47.264 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:47.265 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 msgr2=0x7fde14071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 0x7fde14071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde280b56d0 tx=0x7fde18008040 comp rx=0 tx=0).stop 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 msgr2=0x7fde2815be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fde1c01cca0 tx=0x7fde10005840 comp rx=0 tx=0).stop 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 shutdown_connections 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde28147c70 0x7fde28144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde28153780 0x7fde2815be20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde1406f400 0x7fde14071910 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 --2- 172.21.15.136:0/48255361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde280cef40 0x7fde2814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 >> 172.21.15.136:0/48255361 conn(0x7fde28070f70 msgr2=0x7fde2814a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 shutdown_connections 2024-04-25T11:36:47.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.285+0000 7fde2db59640 1 -- 172.21.15.136:0/48255361 wait complete. 2024-04-25T11:36:47.286 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2024-04-25T11:36:47.286 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T11:36:47.286 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:36:47.303 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T11:36:47.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:47.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.477+0000 7fd29ca43640 1 Processor -- start 2024-04-25T11:36:47.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.477+0000 7fd29ca43640 1 -- start start 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29ca43640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd2980b5060 0x7fd2980c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29ca43640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29815d750 con 0x7fd2980ca290 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29ca43640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29815da30 con 0x7fd29815ab40 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29ca43640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd29815dd10 con 0x7fd2980b5060 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd28ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd2980ca290 0x7fd29815a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46714/0 (socket says 172.21.15.136:46714) 2024-04-25T11:36:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd28ffff640 1 -- 172.21.15.136:0/4175571805 learned_addr learned my addr 172.21.15.136:0/4175571805 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:47.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29659c640 1 --2- 172.21.15.136:0/4175571805 >> v2:172.21.15.157:3300/0 conn(0x7fd2980b5060 0x7fd2980c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645151473 0 0) 0x7fd29815d750 con 0x7fd2980ca290 2024-04-25T11:36:47.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd284003530 con 0x7fd2980ca290 2024-04-25T11:36:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1991280009 0 0) 0x7fd284003530 con 0x7fd2980ca290 2024-04-25T11:36:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 >> v2:172.21.15.157:3300/0 conn(0x7fd2980b5060 msgr2=0x7fd2980c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 --2- 172.21.15.136:0/4175571805 >> v2:172.21.15.157:3300/0 conn(0x7fd2980b5060 0x7fd2980c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 >> v1:172.21.15.157:6789/0 conn(0x7fd29815ab40 legacy=0x7fd298164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29815dff0 con 0x7fd2980ca290 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd280001680 con 0x7fd2980ca290 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 >> v1:172.21.15.136:6789/0 conn(0x7fd2980ca290 legacy=0x7fd29815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2840039c0 0x7fd284003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd284004810 0x7fd284004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 0x7fd28400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd29815dd10 con 0x7fd284005280 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400c200 con 0x7fd284004810 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd28400c400 con 0x7fd2840039c0 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 0x7fd28400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd29659c640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2840039c0 0x7fd284003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 -- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2840039c0 msgr2=0x7fd284003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2840039c0 0x7fd284003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd28ffff640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd284004810 0x7fd284004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 -- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd284004810 msgr2=0x7fd284004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:47.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd284004810 0x7fd284004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.481+0000 7fd296d9d640 1 -- 172.21.15.136:0/4175571805 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd284003530 con 0x7fd284005280 2024-04-25T11:36:47.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd296d9d640 1 -- 172.21.15.136:0/4175571805 reap_dead start 2024-04-25T11:36:47.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd296d9d640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 0x7fd28400b9d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd298068fc0 tx=0x7fd29001c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c422f00192ed7e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290001a90 con 0x7fd284005280 2024-04-25T11:36:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd290003b80 con 0x7fd284005280 2024-04-25T11:36:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd295d9b640 1 -- 172.21.15.136:0/4175571805 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2900050d0 con 0x7fd284005280 2024-04-25T11:36:47.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 -- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 msgr2=0x7fd28400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 0x7fd28400b9d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd298068fc0 tx=0x7fd29001c720 comp rx=0 tx=0).stop 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 -- 172.21.15.136:0/4175571805 shutdown_connections 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 --2- 172.21.15.136:0/4175571805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd284005280 0x7fd28400b9d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 -- 172.21.15.136:0/4175571805 >> 172.21.15.136:0/4175571805 conn(0x7fd298070f70 msgr2=0x7fd298068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 -- 172.21.15.136:0/4175571805 shutdown_connections 2024-04-25T11:36:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.485+0000 7fd29ca43640 1 -- 172.21.15.136:0/4175571805 wait complete. 2024-04-25T11:36:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 Processor -- start 2024-04-25T11:36:47.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- start start 2024-04-25T11:36:47.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd284005280 0x7fd29814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980ccc00 0x7fd298157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd29815d3f0 con 0x7fd29806f070 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd298159f10 con 0x7fd2980ccc00 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd29815d570 con 0x7fd284005280 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47208/0 (socket says 172.21.15.136:47208) 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 -- 172.21.15.136:0/44407564 learned_addr learned my addr 172.21.15.136:0/44407564 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29659c640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd284005280 0x7fd29814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd28ffff640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980ccc00 0x7fd298157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 -- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd284005280 msgr2=0x7fd29814f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd284005280 0x7fd29814f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 -- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980ccc00 msgr2=0x7fd298157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980ccc00 0x7fd298157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 -- 172.21.15.136:0/44407564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd298158480 con 0x7fd29806f070 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd296d9d640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd29001cc00 tx=0x7fd284005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290007dc0 con 0x7fd29806f070 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd290003980 con 0x7fd29806f070 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29806ab00 con 0x7fd29806f070 2024-04-25T11:36:47.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290001a90 con 0x7fd29806f070 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.489+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29815bf00 con 0x7fd29806f070 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd29806a350 con 0x7fd29806f070 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd2900a9870 con 0x7fd29806f070 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd28effd640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 0x7fd278071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fd290129540 con 0x7fd29806f070 2024-04-25T11:36:47.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd29659c640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 0x7fd278071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:47.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.493+0000 7fd29659c640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 0x7fd278071910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd28800f930 tx=0x7fd288008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:47.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.497+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd290016430 con 0x7fd29806f070 2024-04-25T11:36:47.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.729+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2981577f0 con 0x7fd29806f070 2024-04-25T11:36:47.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.729+0000 7fd28effd640 1 -- 172.21.15.136:0/44407564 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7fd2900f5f00 con 0x7fd29806f070 2024-04-25T11:36:47.715 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:47.715 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:47.715 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 msgr2=0x7fd278071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 0x7fd278071910 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd28800f930 tx=0x7fd288008040 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 msgr2=0x7fd2981f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd29001cc00 tx=0x7fd284005840 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 shutdown_connections 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd29806f070 0x7fd2981f59f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980ccc00 0x7fd298157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd27806f400 0x7fd278071910 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 --2- 172.21.15.136:0/44407564 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd284005280 0x7fd29814f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 >> 172.21.15.136:0/44407564 conn(0x7fd298070f70 msgr2=0x7fd298075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 shutdown_connections 2024-04-25T11:36:47.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:47.733+0000 7fd29ca43640 1 -- 172.21.15.136:0/44407564 wait complete. 2024-04-25T11:36:47.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:36:49.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.929+0000 7f9c102db640 1 Processor -- start 2024-04-25T11:36:49.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.929+0000 7f9c102db640 1 -- start start 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c102db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9c080ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c102db640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9c0815a430 0x7f9c0815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c102db640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c0815d750 con 0x7f9c080b5060 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c102db640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9c0815da30 con 0x7f9c0815a430 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c102db640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0815dd10 con 0x7f9c080ca290 2024-04-25T11:36:49.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e077640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9c080b5060 0x7f9c080c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46730/0 (socket says 172.21.15.136:46730) 2024-04-25T11:36:49.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e077640 1 -- 172.21.15.136:0/607369908 learned_addr learned my addr 172.21.15.136:0/607369908 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:49.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d876640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9c080ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331177990 0 0) 0x7f9c0815d750 con 0x7f9c080b5060 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 --2- 172.21.15.136:0/607369908 >> v2:172.21.15.157:3300/0 conn(0x7f9c0815a430 0x7f9c0815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bf4003530 con 0x7f9c080b5060 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1647215622 0 0) 0x7f9bf4003530 con 0x7f9c080b5060 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 msgr2=0x7f9c080ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9c080ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 >> v2:172.21.15.157:3300/0 conn(0x7f9c0815a430 msgr2=0x7f9c0815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 --2- 172.21.15.136:0/607369908 >> v2:172.21.15.157:3300/0 conn(0x7f9c0815a430 0x7f9c0815a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0815dff0 con 0x7f9c080b5060 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f9c04001680 con 0x7f9c080b5060 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 >> v1:172.21.15.136:6789/0 conn(0x7f9c080b5060 legacy=0x7f9c080c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf40039c0 0x7f9bf4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9bf4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf4004d50 0x7f9bf4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0815da30 con 0x7f9bf4004d50 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0815dd10 con 0x7f9bf40039c0 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf400ba50 con 0x7f9c080ca290 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf4004d50 0x7f9bf4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e077640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf40039c0 0x7f9bf4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0d876640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9bf4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 -- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 msgr2=0x7f9bf4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c080ca290 0x7f9bf4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 -- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf40039c0 msgr2=0x7f9bf4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bf40039c0 0x7f9bf4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.933+0000 7f9c0e878640 1 -- 172.21.15.136:0/607369908 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf4003530 con 0x7f9bf4004d50 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c0e878640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf4004d50 0x7f9bf4005160 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009f90 tx=0x7f9c0001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ce96e5dcf2b82f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000b2070 con 0x7f9bf4004d50 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000a5c90 con 0x7f9bf4004d50 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c0d075640 1 -- 172.21.15.136:0/607369908 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00009c10 con 0x7f9bf4004d50 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- 172.21.15.136:0/607369908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf4004d50 msgr2=0x7f9bf4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 --2- 172.21.15.136:0/607369908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bf4004d50 0x7f9bf4005160 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009f90 tx=0x7f9c0001cc80 comp rx=0 tx=0).stop 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c0e878640 1 -- 172.21.15.136:0/607369908 reap_dead start 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- 172.21.15.136:0/607369908 shutdown_connections 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- 172.21.15.136:0/607369908 >> 172.21.15.136:0/607369908 conn(0x7f9c08070f70 msgr2=0x7f9c08068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- 172.21.15.136:0/607369908 shutdown_connections 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- 172.21.15.136:0/607369908 wait complete. 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 Processor -- start 2024-04-25T11:36:49.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.937+0000 7f9c102db640 1 -- start start 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c0814d000 0x7f9c081e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c0814e840 0x7f9c08152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0815ad80 con 0x7f9c0814fdb0 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9c08159f10 con 0x7f9c0814d000 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0815d3f0 con 0x7f9c0814e840 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47222/0 (socket says 172.21.15.136:47222) 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 -- 172.21.15.136:0/960855436 learned_addr learned my addr 172.21.15.136:0/960855436 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0d876640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c0814e840 0x7f9c08152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e077640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c0814d000 0x7f9c081e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 -- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c0814e840 msgr2=0x7f9c08152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c0814e840 0x7f9c08152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 -- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c0814d000 msgr2=0x7f9c081e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:49.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c0814d000 0x7f9c081e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 -- 172.21.15.136:0/960855436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c081e49c0 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c0e878640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9c000b0040 tx=0x7f9c00009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000b2040 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000adb30 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00009c10 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c081e6350 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c081e5aa0 con 0x7f9c0814fdb0 2024-04-25T11:36:49.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.941+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c081e5820 con 0x7f9c0814fdb0 2024-04-25T11:36:49.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.945+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9c000b6410 con 0x7f9c0814fdb0 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.945+0000 7f9bfeffd640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 0x7f9bec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.945+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f9c00129180 con 0x7f9c0814fdb0 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.945+0000 7f9c0e077640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 0x7f9bec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:49.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.945+0000 7f9c0e077640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 0x7f9bec071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9c04000c10 tx=0x7f9c04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:49.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:49.953+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c081e5820 con 0x7f9c0814fdb0 2024-04-25T11:36:50.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.185+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c080cadd0 con 0x7f9c0814fdb0 2024-04-25T11:36:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.185+0000 7f9bfeffd640 1 -- 172.21.15.136:0/960855436 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7f9c000c0060 con 0x7f9c0814fdb0 2024-04-25T11:36:50.169 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:50.169 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:50.170 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 msgr2=0x7f9bec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 0x7f9bec071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9c04000c10 tx=0x7f9c04009040 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 msgr2=0x7f9c081501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9c000b0040 tx=0x7f9c00009750 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 shutdown_connections 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c0814fdb0 0x7f9c081501c0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9bec06f400 0x7f9bec071910 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9c0814e840 0x7f9c08152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 --2- 172.21.15.136:0/960855436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9c0814d000 0x7f9c081e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 >> 172.21.15.136:0/960855436 conn(0x7f9c08070f70 msgr2=0x7f9c080b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 shutdown_connections 2024-04-25T11:36:50.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:50.189+0000 7f9c102db640 1 -- 172.21.15.136:0/960855436 wait complete. 2024-04-25T11:36:50.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:36:52.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:52.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 Processor -- start 2024-04-25T11:36:52.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 -- start start 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb77415c480 0x7fb774167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7740b4480 con 0x7fb7740b5810 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb77415c990 con 0x7fb77415c070 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb77b6f6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb7740b4a90 con 0x7fb77415c480 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb77415c480 0x7fb774167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb77415c480 0x7fb774167920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47224/0 (socket says 172.21.15.136:47224) 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779492640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb7740b5810 0x7fb7740b4f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46746/0 (socket says 172.21.15.136:46746) 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 -- 172.21.15.136:0/3803921224 learned_addr learned my addr 172.21.15.136:0/3803921224 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 -- 172.21.15.136:0/3803921224 >> v1:172.21.15.157:6789/0 conn(0x7fb77415c070 legacy=0x7fb77406d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 -- 172.21.15.136:0/3803921224 >> v1:172.21.15.136:6789/0 conn(0x7fb7740b5810 legacy=0x7fb7740b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.353+0000 7fb779c93640 1 -- 172.21.15.136:0/3803921224 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77415cc70 con 0x7fb77415c480 2024-04-25T11:36:52.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb779c93640 1 --2- 172.21.15.136:0/3803921224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774167920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009e50 tx=0x7fb76c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccf52064ae923b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb76bfff640 1 -- 172.21.15.136:0/3803921224 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b4070 con 0x7fb77415c480 2024-04-25T11:36:52.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb76bfff640 1 -- 172.21.15.136:0/3803921224 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c0af400 con 0x7fb77415c480 2024-04-25T11:36:52.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb76bfff640 1 -- 172.21.15.136:0/3803921224 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0af7d0 con 0x7fb77415c480 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- 172.21.15.136:0/3803921224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 msgr2=0x7fb774167920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/3803921224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774167920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009e50 tx=0x7fb76c01f1e0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- 172.21.15.136:0/3803921224 shutdown_connections 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/3803921224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774167920 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- 172.21.15.136:0/3803921224 >> 172.21.15.136:0/3803921224 conn(0x7fb7740743a0 msgr2=0x7fb7740d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- 172.21.15.136:0/3803921224 shutdown_connections 2024-04-25T11:36:52.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- 172.21.15.136:0/3803921224 wait complete. 2024-04-25T11:36:52.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 Processor -- start 2024-04-25T11:36:52.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.357+0000 7fb77b6f6640 1 -- start start 2024-04-25T11:36:52.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7740b5810 0x7fb7741f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 0x7fb774157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb77406e030 con 0x7fb77415c480 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb77415cf50 con 0x7fb7740b5810 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb77406dc20 con 0x7fb77415c070 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47230/0 (socket says 172.21.15.136:47230) 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 -- 172.21.15.136:0/1714184151 learned_addr learned my addr 172.21.15.136:0/1714184151 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb778c91640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 0x7fb774157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 -- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 msgr2=0x7fb774157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 0x7fb774157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 -- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7740b5810 msgr2=0x7fb7741f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7740b5810 0x7fb7741f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 -- 172.21.15.136:0/1714184151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7741f46b0 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb778c91640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 0x7fb774157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb779c93640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0b2040 tx=0x7fb76c01bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b4040 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c0c0070 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7741f5950 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c009510 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb774157e30 con 0x7fb77415c480 2024-04-25T11:36:52.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.361+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77406a350 con 0x7fb77415c480 2024-04-25T11:36:52.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.365+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb76c0ba660 con 0x7fb77415c480 2024-04-25T11:36:52.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.365+0000 7fb769ffb640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 0x7fb758071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:52.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.365+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7fb76c12e780 con 0x7fb77415c480 2024-04-25T11:36:52.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.365+0000 7fb779492640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 0x7fb758071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:52.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.365+0000 7fb779492640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 0x7fb758071910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b4650 tx=0x7fb7700079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:52.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.369+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb7741f46b0 con 0x7fb77415c480 2024-04-25T11:36:52.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.593+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7740cd5f0 con 0x7fb77415c480 2024-04-25T11:36:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb769ffb640 1 -- 172.21.15.136:0/1714184151 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7fb76c018d40 con 0x7fb77415c480 2024-04-25T11:36:52.578 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:52.578 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 msgr2=0x7fb758071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 0x7fb758071910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7740b4650 tx=0x7fb7700079b0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 msgr2=0x7fb774153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0b2040 tx=0x7fb76c01bf80 comp rx=0 tx=0).stop 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 shutdown_connections 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77415c480 0x7fb774153c00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb75806f400 0x7fb758071910 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb77415c070 0x7fb774157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 --2- 172.21.15.136:0/1714184151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7740b5810 0x7fb7741f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 >> 172.21.15.136:0/1714184151 conn(0x7fb7740743a0 msgr2=0x7fb7740cbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 shutdown_connections 2024-04-25T11:36:52.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:52.597+0000 7fb77b6f6640 1 -- 172.21.15.136:0/1714184151 wait complete. 2024-04-25T11:36:52.598 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:36:53.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:36:53.976 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:36:54.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:54.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.789+0000 7f976cd78640 1 Processor -- start 2024-04-25T11:36:54.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.789+0000 7f976cd78640 1 -- start start 2024-04-25T11:36:54.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976cd78640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f97680c8cd0 0x7f976814f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976cd78640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f97680ce260 0x7f97680ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976cd78640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97680cbad0 con 0x7f97680ca010 2024-04-25T11:36:54.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976cd78640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f97680c8550 con 0x7f97680ce260 2024-04-25T11:36:54.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976cd78640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f97680c7c20 con 0x7f97680c8cd0 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f97680ce260 0x7f97680ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f97680ce260 0x7f97680ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38992/0 (socket says 172.21.15.136:38992) 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 -- 172.21.15.136:0/1477668603 learned_addr learned my addr 172.21.15.136:0/1477668603 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f976659c640 1 --2- 172.21.15.136:0/1477668603 >> v2:172.21.15.157:3300/0 conn(0x7f97680c8cd0 0x7f976814f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 -- 172.21.15.136:0/1477668603 >> v2:172.21.15.157:3300/0 conn(0x7f97680c8cd0 msgr2=0x7f976814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 --2- 172.21.15.136:0/1477668603 >> v2:172.21.15.157:3300/0 conn(0x7f97680c8cd0 0x7f976814f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 -- 172.21.15.136:0/1477668603 >> v1:172.21.15.157:6789/0 conn(0x7f97680ca010 legacy=0x7f97680ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:36:54.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.793+0000 7f9765d9b640 1 -- 172.21.15.136:0/1477668603 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97680c7f00 con 0x7f97680ce260 2024-04-25T11:36:54.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f9765d9b640 1 --2- 172.21.15.136:0/1477668603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f97680ce670 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f97680c86d0 tx=0x7f975c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efc2b8cbf2e81aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:54.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976559a640 1 -- 172.21.15.136:0/1477668603 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c0b5070 con 0x7f97680ce260 2024-04-25T11:36:54.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976559a640 1 -- 172.21.15.136:0/1477668603 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c0a89d0 con 0x7f97680ce260 2024-04-25T11:36:54.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976559a640 1 -- 172.21.15.136:0/1477668603 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c0a8da0 con 0x7f97680ce260 2024-04-25T11:36:54.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 -- 172.21.15.136:0/1477668603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 msgr2=0x7f97680ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 --2- 172.21.15.136:0/1477668603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f97680ce670 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f97680c86d0 tx=0x7f975c01ecd0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 -- 172.21.15.136:0/1477668603 shutdown_connections 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 --2- 172.21.15.136:0/1477668603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f97680ce670 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 --2- 172.21.15.136:0/1477668603 >> v2:172.21.15.157:3300/0 conn(0x7f97680c8cd0 0x7f976814f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 -- 172.21.15.136:0/1477668603 >> 172.21.15.136:0/1477668603 conn(0x7f976806f9d0 msgr2=0x7f976806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 -- 172.21.15.136:0/1477668603 shutdown_connections 2024-04-25T11:36:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.797+0000 7f976cd78640 1 -- 172.21.15.136:0/1477668603 wait complete. 2024-04-25T11:36:54.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 Processor -- start 2024-04-25T11:36:54.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- start start 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 0x7f97681f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97680ca010 0x7f9768154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f97680cb720 con 0x7f97680ce260 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f976814fbc0 con 0x7f97680c8cd0 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9768154c10 con 0x7f97680ca010 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39004/0 (socket says 172.21.15.136:39004) 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 -- 172.21.15.136:0/620247258 learned_addr learned my addr 172.21.15.136:0/620247258 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:54.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976659c640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 0x7f97681f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9765d9b640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97680ca010 0x7f9768154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 -- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97680ca010 msgr2=0x7f9768154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97680ca010 0x7f9768154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 -- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 msgr2=0x7f97681f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 0x7f97681f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 -- 172.21.15.136:0/620247258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976814b5f0 con 0x7f97680ce260 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976659c640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 0x7f97681f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9766d9d640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f976000b970 tx=0x7f976000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760019070 con 0x7f97680ce260 2024-04-25T11:36:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f976000cb40 con 0x7f97680ce260 2024-04-25T11:36:54.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9768154740 con 0x7f97680ce260 2024-04-25T11:36:54.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760014440 con 0x7f97680ce260 2024-04-25T11:36:54.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.801+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976814b030 con 0x7f97680ce260 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976806a350 con 0x7f97680ce260 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f976001c070 con 0x7f97680ce260 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f9756ffd640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 0x7f9748071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f976008f5b0 con 0x7f97680ce260 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f976659c640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 0x7f9748071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.805+0000 7f976659c640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 0x7f9748071910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f975800b9f0 tx=0x7f9758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:54.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:54.809+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f976005bdf0 con 0x7f97680ce260 2024-04-25T11:36:55.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.053+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9768156790 con 0x7f97680ce260 2024-04-25T11:36:55.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.053+0000 7f9756ffd640 1 -- 172.21.15.136:0/620247258 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7f9760062e20 con 0x7f97680ce260 2024-04-25T11:36:55.036 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:55.036 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:55.036 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 msgr2=0x7f9748071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 0x7f9748071910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f975800b9f0 tx=0x7f9758008040 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 msgr2=0x7f976814dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f976000b970 tx=0x7f976000be30 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 shutdown_connections 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97680ce260 0x7f976814dc40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f974806f400 0x7f9748071910 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97680ca010 0x7f9768154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 --2- 172.21.15.136:0/620247258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97680c8cd0 0x7f97681f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 >> 172.21.15.136:0/620247258 conn(0x7f976806f9d0 msgr2=0x7f97680c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:55.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 shutdown_connections 2024-04-25T11:36:55.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:55.057+0000 7f976cd78640 1 -- 172.21.15.136:0/620247258 wait complete. 2024-04-25T11:36:55.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:36:57.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:57.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.253+0000 7f073958b640 1 Processor -- start 2024-04-25T11:36:57.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.253+0000 7f073958b640 1 -- start start 2024-04-25T11:36:57.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f073958b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0734160370 0x7f0734167a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:57.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f073958b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073415c6f0 con 0x7f073415ff60 2024-04-25T11:36:57.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f073958b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073415c990 con 0x7f073415c0d0 2024-04-25T11:36:57.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f073958b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f07340c6310 con 0x7f0734160370 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0734160370 0x7f0734167a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f072bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f073415ff60 0x7f07340ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55760/0 (socket says 172.21.15.136:55760) 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 --2- 172.21.15.136:0/3429100106 >> v2:172.21.15.136:3300/0 conn(0x7f0734160370 0x7f0734167a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39014/0 (socket says 172.21.15.136:39014) 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f072bfff640 1 -- 172.21.15.136:0/3429100106 learned_addr learned my addr 172.21.15.136:0/3429100106 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 -- 172.21.15.136:0/3429100106 >> v1:172.21.15.157:6789/0 conn(0x7f073415c0d0 legacy=0x7f07340cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 -- 172.21.15.136:0/3429100106 >> v1:172.21.15.136:6789/0 conn(0x7f073415ff60 legacy=0x7f07340ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 -- 172.21.15.136:0/3429100106 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073415cc30 con 0x7f0734160370 2024-04-25T11:36:57.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07327fc640 1 -- 172.21.15.136:0/3429100106 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217093746 0 0) 0x7f073415c6f0 con 0x7f073415ff60 2024-04-25T11:36:57.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07337fe640 1 --2- 172.21.15.136:0/3429100106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0734160370 0x7f0734167a60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f072c009d90 tx=0x7f072c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d6cf146410bc2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:57.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07327fc640 1 -- 172.21.15.136:0/3429100106 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c0b2070 con 0x7f0734160370 2024-04-25T11:36:57.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07327fc640 1 -- 172.21.15.136:0/3429100106 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072c0a5920 con 0x7f0734160370 2024-04-25T11:36:57.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.257+0000 7f07327fc640 1 -- 172.21.15.136:0/3429100106 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c0a5cf0 con 0x7f0734160370 2024-04-25T11:36:57.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- 172.21.15.136:0/3429100106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0734160370 msgr2=0x7f0734167a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 --2- 172.21.15.136:0/3429100106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0734160370 0x7f0734167a60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f072c009d90 tx=0x7f072c01c970 comp rx=0 tx=0).stop 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- 172.21.15.136:0/3429100106 shutdown_connections 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 --2- 172.21.15.136:0/3429100106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0734160370 0x7f0734167a60 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- 172.21.15.136:0/3429100106 >> 172.21.15.136:0/3429100106 conn(0x7f073406f9d0 msgr2=0x7f073406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- 172.21.15.136:0/3429100106 shutdown_connections 2024-04-25T11:36:57.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- 172.21.15.136:0/3429100106 wait complete. 2024-04-25T11:36:57.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 Processor -- start 2024-04-25T11:36:57.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- start start 2024-04-25T11:36:57.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 0x7f07340c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:57.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:57.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0734160370 0x7f0734144bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f07340c9170 con 0x7f073415ff60 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07340cabc0 con 0x7f073415c0d0 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f073958b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f07340c7c10 con 0x7f0734160370 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39026/0 (socket says 172.21.15.136:39026) 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 -- 172.21.15.136:0/2987156921 learned_addr learned my addr 172.21.15.136:0/2987156921 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f0732ffd640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 0x7f07340c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f07337fe640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0734160370 0x7f0734144bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 -- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0734160370 msgr2=0x7f0734144bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0734160370 0x7f0734144bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 -- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 msgr2=0x7f07340c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 0x7f07340c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f072bfff640 1 -- 172.21.15.136:0/2987156921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07340c85e0 con 0x7f073415ff60 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.261+0000 7f0732ffd640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 0x7f07340c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:57.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072bfff640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c410 tx=0x7f071c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:57.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c0074e0 con 0x7f073415ff60 2024-04-25T11:36:57.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071c0047f0 con 0x7f073415ff60 2024-04-25T11:36:57.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0734153970 con 0x7f073415ff60 2024-04-25T11:36:57.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c007680 con 0x7f073415ff60 2024-04-25T11:36:57.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0734153c30 con 0x7f073415ff60 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073406a350 con 0x7f073415ff60 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f071c011070 con 0x7f073415ff60 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 0x7f0714071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7423+0+0 (secure 0 0 0) 0x7f071c08fc50 con 0x7f073415ff60 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.265+0000 7f0732ffd640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 0x7f0714071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:57.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.269+0000 7f0732ffd640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 0x7f0714071910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0724000f70 tx=0x7f0724009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:57.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.269+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f071c05c490 con 0x7f073415ff60 2024-04-25T11:36:57.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.513+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0734159d20 con 0x7f073415ff60 2024-04-25T11:36:57.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.513+0000 7f072b7fe640 1 -- 172.21.15.136:0/2987156921 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6931 (secure 0 0 0) 0x7f071c05bd90 con 0x7f073415ff60 2024-04-25T11:36:57.496 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:57.496 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6246,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:19.340324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":5249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5249":{"gid":5249,"name":"b","rank":4,"incarnation":62,"state":"up:active","state_seq":39,"addr":"172.21.15.157:6837/3576995854","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3576995854},{"type":"v1","addr":"172.21.15.157:6837","nonce":3576995854}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356,5249]},"id":1}]} 2024-04-25T11:36:57.496 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 67 2024-04-25T11:36:57.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 msgr2=0x7f0714071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 0x7f0714071910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0724000f70 tx=0x7f0724009040 comp rx=0 tx=0).stop 2024-04-25T11:36:57.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 msgr2=0x7f0734147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:57.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c410 tx=0x7f071c00c8d0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 shutdown_connections 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f071406f400 0x7f0714071910 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0734160370 0x7f0734144bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f073415ff60 0x7f0734147e90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 --2- 172.21.15.136:0/2987156921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f073415c0d0 0x7f07340c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 >> 172.21.15.136:0/2987156921 conn(0x7f073406f9d0 msgr2=0x7f073415ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 shutdown_connections 2024-04-25T11:36:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:57.517+0000 7f073958b640 1 -- 172.21.15.136:0/2987156921 wait complete. 2024-04-25T11:36:57.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:36:59.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:36:59.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 Processor -- start 2024-04-25T11:36:59.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 -- start start 2024-04-25T11:36:59.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 0x7fe2d414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:59.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d40cbad0 con 0x7fe2d40ca720 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d40c7c20 con 0x7fe2d40ce260 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dc6ad640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d40c8550 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dac4a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe2d40ca720 0x7fe2d41637f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55764/0 (socket says 172.21.15.136:55764) 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2dac4a640 1 -- 172.21.15.136:0/3760697239 learned_addr learned my addr 172.21.15.136:0/3760697239 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.717+0000 7fe2da449640 1 --2- 172.21.15.136:0/3760697239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 0x7fe2d414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778594914 0 0) 0x7fe2d40cbad0 con 0x7fe2d40ca720 2024-04-25T11:36:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2c0005740 con 0x7fe2d40ca720 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317104933 0 0) 0x7fe2d40c7c20 con 0x7fe2d40ce260 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d40cbad0 con 0x7fe2d40ce260 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2840247814 0 0) 0x7fe2c0005740 con 0x7fe2d40ca720 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2da449640 1 -- 172.21.15.136:0/3760697239 >> v1:172.21.15.157:6789/0 conn(0x7fe2d40ce260 legacy=0x7fe2d40ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2da449640 1 -- 172.21.15.136:0/3760697239 >> v1:172.21.15.136:6789/0 conn(0x7fe2d40ca720 legacy=0x7fe2d41637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2da449640 1 -- 172.21.15.136:0/3760697239 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d40c7f00 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2da449640 1 --2- 172.21.15.136:0/3760697239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 0x7fe2d414f680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00097f0 tx=0x7fe2d001ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03674eacd95d577 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:59.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.721+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00b5070 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d00a8670 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2d9447640 1 -- 172.21.15.136:0/3760697239 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc001a00 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/3760697239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 msgr2=0x7fe2d414f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/3760697239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 0x7fe2d414f680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00097f0 tx=0x7fe2d001ebf0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/3760697239 shutdown_connections 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/3760697239 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40c8cd0 0x7fe2d414f680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/3760697239 >> 172.21.15.136:0/3760697239 conn(0x7fe2d406f9d0 msgr2=0x7fe2d406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/3760697239 shutdown_connections 2024-04-25T11:36:59.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/3760697239 wait complete. 2024-04-25T11:36:59.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 Processor -- start 2024-04-25T11:36:59.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.725+0000 7fe2dc6ad640 1 -- start start 2024-04-25T11:36:59.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:59.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d40ca720 0x7fe2d415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 0x7fe2d414e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d40cb720 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d414fbc0 con 0x7fe2d40ca720 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d414eb40 con 0x7fe2d40ce260 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39028/0 (socket says 172.21.15.136:39028) 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 -- 172.21.15.136:0/1424489525 learned_addr learned my addr 172.21.15.136:0/1424489525 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dac4a640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 0x7fe2d414e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2d9c48640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d40ca720 0x7fe2d415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:59.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 -- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 msgr2=0x7fe2d414e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 0x7fe2d414e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 -- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d40ca720 msgr2=0x7fe2d415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d40ca720 0x7fe2d415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 -- 172.21.15.136:0/1424489525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d40cd8a0 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dac4a640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 0x7fe2d414e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2da449640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00b3040 tx=0x7fe2d001b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00bf070 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d00a8a20 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00b5040 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d414e670 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.729+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d41e3c70 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d41e4330 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe2d00c2070 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2caffd640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 0x7fe2b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:36:59.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fe2d012bff0 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2d9c48640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 0x7fe2b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:36:59.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.733+0000 7fe2d9c48640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 0x7fe2b8071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2d414dd30 tx=0x7fe2c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:36:59.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.737+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe2d41e4330 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2d406a350 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2caffd640 1 -- 172.21.15.136:0/1424489525 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6098 (secure 0 0 0) 0x7fe2d40cd8a0 con 0x7fe2d40c8cd0 2024-04-25T11:36:59.976 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:36:59.976 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:58.974597+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:36:59.976 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 68 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 msgr2=0x7fe2b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 0x7fe2b8071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2d414dd30 tx=0x7fe2c4009040 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 msgr2=0x7fe2d41410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00b3040 tx=0x7fe2d001b990 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 shutdown_connections 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2b806f400 0x7fe2b8071910 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2d40ce260 0x7fe2d414e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2d40ca720 0x7fe2d415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 --2- 172.21.15.136:0/1424489525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2d40c8cd0 0x7fe2d41410c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.993+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 >> 172.21.15.136:0/1424489525 conn(0x7fe2d406f9d0 msgr2=0x7fe2d414b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:36:59.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.997+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 shutdown_connections 2024-04-25T11:36:59.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:36:59.997+0000 7fe2dc6ad640 1 -- 172.21.15.136:0/1424489525 wait complete. 2024-04-25T11:36:59.997 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-25T11:36:59.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:36:59.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:00.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 Processor -- start 2024-04-25T11:37:00.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 -- start start 2024-04-25T11:37:00.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7ec154060 0x7ff7ec0c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7ec15a5f0 0x7ff7ec0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ec0c8fd0 con 0x7ff7ec154060 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ec0c8d40 con 0x7ff7ec15a5f0 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7f0c98640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ec0c3880 con 0x7ff7ec15aa00 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7ea59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7ec154060 0x7ff7ec0c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7ea59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7ec154060 0x7ff7ec0c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39032/0 (socket says 172.21.15.136:39032) 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7ea59c640 1 -- 172.21.15.136:0/1546410828 learned_addr learned my addr 172.21.15.136:0/1546410828 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7ead9d640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.181+0000 7ff7e9d9b640 1 --2- 172.21.15.136:0/1546410828 >> v2:172.21.15.157:3300/0 conn(0x7ff7ec15a5f0 0x7ff7ec0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 -- 172.21.15.136:0/1546410828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 msgr2=0x7ff7ec0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec0c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 -- 172.21.15.136:0/1546410828 >> v2:172.21.15.157:3300/0 conn(0x7ff7ec15a5f0 msgr2=0x7ff7ec0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:00.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/1546410828 >> v2:172.21.15.157:3300/0 conn(0x7ff7ec15a5f0 0x7ff7ec0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 -- 172.21.15.136:0/1546410828 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec06d9b0 con 0x7ff7ec154060 2024-04-25T11:37:00.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec0c8800 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06db80 tx=0x7ff7dc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eaeb0f6c30ef41e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7e959a640 1 -- 172.21.15.136:0/1546410828 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b5070 con 0x7ff7ec154060 2024-04-25T11:37:00.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7e959a640 1 -- 172.21.15.136:0/1546410828 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0a89d0 con 0x7ff7ec154060 2024-04-25T11:37:00.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7e959a640 1 -- 172.21.15.136:0/1546410828 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0a8da0 con 0x7ff7ec154060 2024-04-25T11:37:00.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 -- 172.21.15.136:0/1546410828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 msgr2=0x7ff7ec0c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec0c8800 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06db80 tx=0x7ff7dc01ed10 comp rx=0 tx=0).stop 2024-04-25T11:37:00.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 -- 172.21.15.136:0/1546410828 shutdown_connections 2024-04-25T11:37:00.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec0c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/1546410828 >> v2:172.21.15.157:3300/0 conn(0x7ff7ec15a5f0 0x7ff7ec0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/1546410828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec0c8800 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 -- 172.21.15.136:0/1546410828 >> 172.21.15.136:0/1546410828 conn(0x7ff7ec0ca5a0 msgr2=0x7ff7ec0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 -- 172.21.15.136:0/1546410828 shutdown_connections 2024-04-25T11:37:00.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.185+0000 7ff7f0c98640 1 -- 172.21.15.136:0/1546410828 wait complete. 2024-04-25T11:37:00.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 Processor -- start 2024-04-25T11:37:00.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 -- start start 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 0x7ff7ec1486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec06df30 con 0x7ff7ec154060 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec154470 con 0x7ff7ec15a5f0 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ec15a0d0 con 0x7ff7ec15aa00 2024-04-25T11:37:00.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39034/0 (socket says 172.21.15.136:39034) 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 -- 172.21.15.136:0/3490025070 learned_addr learned my addr 172.21.15.136:0/3490025070 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ead9d640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7e9d9b640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 0x7ff7ec1486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 -- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 msgr2=0x7ff7ec145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec145150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 -- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 msgr2=0x7ff7ec1486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 0x7ff7ec1486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 -- 172.21.15.136:0/3490025070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec073b80 con 0x7ff7ec154060 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7e9d9b640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 0x7ff7ec1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7ea59c640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b3040 tx=0x7ff7dc01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b5070 con 0x7ff7ec154060 2024-04-25T11:37:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0b0a30 con 0x7ff7ec154060 2024-04-25T11:37:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b0e20 con 0x7ff7ec154060 2024-04-25T11:37:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.189+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec1f67b0 con 0x7ff7ec154060 2024-04-25T11:37:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec1469c0 con 0x7ff7ec154060 2024-04-25T11:37:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec06a350 con 0x7ff7ec154060 2024-04-25T11:37:00.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff7dc0c2060 con 0x7ff7ec154060 2024-04-25T11:37:00.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7daffd640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 0x7ff7cc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:00.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7ff7dc12c9e0 con 0x7ff7ec154060 2024-04-25T11:37:00.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7e9d9b640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 0x7ff7cc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:00.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.193+0000 7ff7e9d9b640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 0x7ff7cc071940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0cd990 tx=0x7ff7e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:00.183 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.201+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff7dc018870 con 0x7ff7ec154060 2024-04-25T11:37:00.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7ec0c85e0 con 0x7ff7ec154060 2024-04-25T11:37:00.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7daffd640 1 -- 172.21.15.136:0/3490025070 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6098 (secure 0 0 0) 0x7ff7dc0f9250 con 0x7ff7ec154060 2024-04-25T11:37:00.444 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:00.444 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:58.974597+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:37:00.444 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 68 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 msgr2=0x7ff7cc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 0x7ff7cc071940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0cd990 tx=0x7ff7e0008040 comp rx=0 tx=0).stop 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 msgr2=0x7ff7ec1f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b3040 tx=0x7ff7dc01bab0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 shutdown_connections 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff7cc06f430 0x7ff7cc071940 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ec15aa00 0x7ff7ec145150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.461+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ec15a5f0 0x7ff7ec1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.465+0000 7ff7f0c98640 1 --2- 172.21.15.136:0/3490025070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7ec154060 0x7ff7ec1f6270 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.465+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 >> 172.21.15.136:0/3490025070 conn(0x7ff7ec0ca5a0 msgr2=0x7ff7ec1596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.465+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 shutdown_connections 2024-04-25T11:37:00.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:00.465+0000 7ff7f0c98640 1 -- 172.21.15.136:0/3490025070 wait complete. 2024-04-25T11:37:00.464 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 68, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 68, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:36:58.974597+0000", "qdb_cluster": [ 5356, 5385, 5449, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:37:02.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:02.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.653+0000 7f1257fff640 1 Processor -- start 2024-04-25T11:37:02.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.653+0000 7f1257fff640 1 -- start start 2024-04-25T11:37:02.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 0x7f125806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1258159950 0x7f12580c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12580c5580 0x7f12580c5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f12580c7660 con 0x7f12580c5580 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f12580c7920 con 0x7f1258159950 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1257fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12580c7c00 con 0x7f1258159540 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12580c5580 0x7f12580c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12580c5580 0x7f12580c5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39044/0 (socket says 172.21.15.136:39044) 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 -- 172.21.15.136:0/635050358 learned_addr learned my addr 172.21.15.136:0/635050358 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12567fc640 1 --2- 172.21.15.136:0/635050358 >> v2:172.21.15.157:3300/0 conn(0x7f1258159950 0x7f12580c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1256ffd640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 0x7f125806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 -- 172.21.15.136:0/635050358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 msgr2=0x7f125806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 0x7f125806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 -- 172.21.15.136:0/635050358 >> v2:172.21.15.157:3300/0 conn(0x7f1258159950 msgr2=0x7f12580c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 --2- 172.21.15.136:0/635050358 >> v2:172.21.15.157:3300/0 conn(0x7f1258159950 0x7f12580c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 -- 172.21.15.136:0/635050358 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580c7ee0 con 0x7f12580c5580 2024-04-25T11:37:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1256ffd640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 0x7f125806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:02.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f12577fe640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c5990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f12580ceb30 tx=0x7f125001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20b845d3c1393e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:02.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1255ffb640 1 -- 172.21.15.136:0/635050358 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500a5550 con 0x7f12580c5580 2024-04-25T11:37:02.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1255ffb640 1 -- 172.21.15.136:0/635050358 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12500a56f0 con 0x7f12580c5580 2024-04-25T11:37:02.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.657+0000 7f1255ffb640 1 -- 172.21.15.136:0/635050358 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12500a5ac0 con 0x7f12580c5580 2024-04-25T11:37:02.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- 172.21.15.136:0/635050358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 msgr2=0x7f12580c5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:02.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c5990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f12580ceb30 tx=0x7f125001c8d0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- 172.21.15.136:0/635050358 shutdown_connections 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c5990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 --2- 172.21.15.136:0/635050358 >> v2:172.21.15.157:3300/0 conn(0x7f1258159950 0x7f12580c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 --2- 172.21.15.136:0/635050358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159540 0x7f125806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- 172.21.15.136:0/635050358 >> 172.21.15.136:0/635050358 conn(0x7f12580cdb40 msgr2=0x7f12580cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- 172.21.15.136:0/635050358 shutdown_connections 2024-04-25T11:37:02.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- 172.21.15.136:0/635050358 wait complete. 2024-04-25T11:37:02.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 Processor -- start 2024-04-25T11:37:02.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 -- start start 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.661+0000 7f1257fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1258159540 0x7f125813df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 0x7f12581f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f12580c73c0 con 0x7f12580c5580 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12580c7240 con 0x7f1258159540 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1258074210 con 0x7f1258159950 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39060/0 (socket says 172.21.15.136:39060) 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 -- 172.21.15.136:0/1692618025 learned_addr learned my addr 172.21.15.136:0/1692618025 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:02.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f12567fc640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1258159540 0x7f125813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f12577fe640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 0x7f12581f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 -- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 msgr2=0x7f12581f8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 0x7f12581f8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 -- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1258159540 msgr2=0x7f125813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1258159540 0x7f125813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 -- 172.21.15.136:0/1692618025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12580cb3f0 con 0x7f12580c5580 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f12577fe640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 0x7f12581f8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1256ffd640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f124800c820 tx=0x7f124800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248015070 con 0x7f12580c5580 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12480079d0 con 0x7f12580c5580 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248007dc0 con 0x7f12580c5580 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125814d460 con 0x7f12580c5580 2024-04-25T11:37:02.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12580c8230 con 0x7f12580c5580 2024-04-25T11:37:02.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125806a2e0 con 0x7f12580c5580 2024-04-25T11:37:02.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1248018070 con 0x7f12580c5580 2024-04-25T11:37:02.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 0x7f1244071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:02.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.665+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f124808f850 con 0x7f12580c5580 2024-04-25T11:37:02.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.669+0000 7f12567fc640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 0x7f1244071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:02.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.669+0000 7f12567fc640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 0x7f1244071910 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f125814df50 tx=0x7f124c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:02.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.673+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f124805c030 con 0x7f12580c5580 2024-04-25T11:37:02.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.917+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f125806c9f0 con 0x7f12580c5580 2024-04-25T11:37:02.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.917+0000 7f123f7fe640 1 -- 172.21.15.136:0/1692618025 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6098 (secure 0 0 0) 0x7f124805b930 con 0x7f12580c5580 2024-04-25T11:37:02.900 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:02.900 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:36:58.974597+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:37:02.901 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 68 2024-04-25T11:37:02.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 msgr2=0x7f1244071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 0x7f1244071910 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f125814df50 tx=0x7f124c008040 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 msgr2=0x7f12580c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f124800c820 tx=0x7f124800cce0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 shutdown_connections 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f124406f400 0x7f1244071910 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1258159950 0x7f12581f8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1258159540 0x7f125813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 --2- 172.21.15.136:0/1692618025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12580c5580 0x7f12580c8d10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 >> 172.21.15.136:0/1692618025 conn(0x7f12580cdb40 msgr2=0x7f125806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 shutdown_connections 2024-04-25T11:37:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:02.921+0000 7f1257fff640 1 -- 172.21.15.136:0/1692618025 wait complete. 2024-04-25T11:37:02.923 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:04.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:05.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.105+0000 7f69327e5640 1 Processor -- start 2024-04-25T11:37:05.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.105+0000 7f69327e5640 1 -- start start 2024-04-25T11:37:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f69327e5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f692c15aae0 0x7f692c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f69327e5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692c0c9af0 con 0x7f692c0c79e0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f69327e5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692c0cb220 con 0x7f692c0c96e0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f69327e5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f692c06cc10 con 0x7f692c15aae0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f692c0c79e0 0x7f692c15a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41120/0 (socket says 172.21.15.136:41120) 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 -- 172.21.15.136:0/2571258481 learned_addr learned my addr 172.21.15.136:0/2571258481 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f6930d82640 1 --2- 172.21.15.136:0/2571258481 >> v2:172.21.15.157:3300/0 conn(0x7f692c15aae0 0x7f692c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112336886 0 0) 0x7f692c0c9af0 con 0x7f692c0c79e0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6918003530 con 0x7f692c0c79e0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540229855 0 0) 0x7f6918003530 con 0x7f692c0c79e0 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 >> v2:172.21.15.157:3300/0 conn(0x7f692c15aae0 msgr2=0x7f692c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 --2- 172.21.15.136:0/2571258481 >> v2:172.21.15.157:3300/0 conn(0x7f692c15aae0 0x7f692c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 >> v1:172.21.15.157:6789/0 conn(0x7f692c0c96e0 legacy=0x7f692c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c0cb500 con 0x7f692c0c79e0 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f6914001680 con 0x7f692c0c79e0 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 >> v1:172.21.15.136:6789/0 conn(0x7f692c0c79e0 legacy=0x7f692c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69180039c0 0x7f6918003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 0x7f6918004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6918005280 0x7f691800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f692c06cc10 con 0x7f6918004810 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f691800c1a0 con 0x7f69180039c0 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f69180107b0 con 0x7f6918005280 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 0x7f6918004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f6930d82640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6918005280 0x7f691800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692bfff640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69180039c0 0x7f6918003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 -- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6918005280 msgr2=0x7f691800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6918005280 0x7f691800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 -- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69180039c0 msgr2=0x7f6918003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69180039c0 0x7f6918003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f692b7fe640 1 -- 172.21.15.136:0/2571258481 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6918003530 con 0x7f6918004810 2024-04-25T11:37:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.109+0000 7f6930d82640 1 -- 172.21.15.136:0/2571258481 reap_dead start 2024-04-25T11:37:05.096 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f692b7fe640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 0x7f6918004c20 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f691400c1c0 tx=0x7f691401f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220c43081dd91e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914007d00 con 0x7f6918004810 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6914005b00 con 0x7f6918004810 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f692affd640 1 -- 172.21.15.136:0/2571258481 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914007070 con 0x7f6918004810 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 -- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 msgr2=0x7f6918004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 0x7f6918004c20 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f691400c1c0 tx=0x7f691401f350 comp rx=0 tx=0).stop 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 -- 172.21.15.136:0/2571258481 shutdown_connections 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 --2- 172.21.15.136:0/2571258481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6918004810 0x7f6918004c20 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 -- 172.21.15.136:0/2571258481 >> 172.21.15.136:0/2571258481 conn(0x7f692c06f9d0 msgr2=0x7f692c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 -- 172.21.15.136:0/2571258481 shutdown_connections 2024-04-25T11:37:05.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.113+0000 7f69327e5640 1 -- 172.21.15.136:0/2571258481 wait complete. 2024-04-25T11:37:05.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 Processor -- start 2024-04-25T11:37:05.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 -- start start 2024-04-25T11:37:05.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6918004810 0x7f692c1f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f692c1499a0 0x7f692c152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f692c0ced40 con 0x7f692c149590 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f692c0cf0f0 con 0x7f6918004810 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f69327e5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f692c1ee370 con 0x7f692c1499a0 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38172/0 (socket says 172.21.15.136:38172) 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 -- 172.21.15.136:0/3684868814 learned_addr learned my addr 172.21.15.136:0/3684868814 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f6930d82640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f692c1499a0 0x7f692c152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 -- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f692c1499a0 msgr2=0x7f692c152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692bfff640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6918004810 0x7f692c1f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f692c1499a0 0x7f692c152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 -- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6918004810 msgr2=0x7f692c1f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6918004810 0x7f692c1f1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 -- 172.21.15.136:0/3684868814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c1f2490 con 0x7f692c149590 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f692b7fe640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6914001980 tx=0x7f6918005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69140047f0 con 0x7f692c149590 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6914004990 con 0x7f692c149590 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.117+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914004d40 con 0x7f692c149590 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c159020 con 0x7f692c149590 2024-04-25T11:37:05.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c1f1750 con 0x7f692c149590 2024-04-25T11:37:05.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692c06a350 con 0x7f692c149590 2024-04-25T11:37:05.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f69140070e0 con 0x7f692c149590 2024-04-25T11:37:05.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f6928ff9640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 0x7f690c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:05.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f691412cef0 con 0x7f692c149590 2024-04-25T11:37:05.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.121+0000 7f692bfff640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 0x7f690c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:05.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.125+0000 7f692bfff640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 0x7f690c071910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f691c0015a0 tx=0x7f691c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:05.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.125+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f69140c2030 con 0x7f692c149590 2024-04-25T11:37:05.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f692c1579a0 con 0x7f692c149590 2024-04-25T11:37:05.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f6928ff9640 1 -- 172.21.15.136:0/3684868814 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6100 (secure 0 0 0) 0x7f6914019060 con 0x7f692c149590 2024-04-25T11:37:05.331 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:05.331 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:03.988268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:resolve","state_seq":12,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:05.331 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 69 2024-04-25T11:37:05.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 msgr2=0x7f690c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 0x7f690c071910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f691c0015a0 tx=0x7f691c009040 comp rx=0 tx=0).stop 2024-04-25T11:37:05.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 msgr2=0x7f692c151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6914001980 tx=0x7f6918005840 comp rx=0 tx=0).stop 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 shutdown_connections 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f690c06f400 0x7f690c071910 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f692c1499a0 0x7f692c152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f692c149590 0x7f692c151b40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 --2- 172.21.15.136:0/3684868814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6918004810 0x7f692c1f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 >> 172.21.15.136:0/3684868814 conn(0x7f692c06f9d0 msgr2=0x7f692c06d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 shutdown_connections 2024-04-25T11:37:05.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:05.348+0000 7f69327e5640 1 -- 172.21.15.136:0/3684868814 wait complete. 2024-04-25T11:37:05.351 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:07.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:07.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.540+0000 7fbb69321640 1 Processor -- start 2024-04-25T11:37:07.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.540+0000 7fbb69321640 1 -- start start 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb69321640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb640ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb69321640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb640ccb80 con 0x7fbb64156330 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb69321640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb64156740 con 0x7fbb640cdee0 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb69321640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb641602f0 con 0x7fbb640b5ba0 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb62ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbb64156330 0x7fbb6415cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41130/0 (socket says 172.21.15.136:41130) 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb62ffd640 1 -- 172.21.15.136:0/2464623474 learned_addr learned my addr 172.21.15.136:0/2464623474 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb640ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323702965 0 0) 0x7fbb640ccb80 con 0x7fbb64156330 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb4c003530 con 0x7fbb64156330 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2762358527 0 0) 0x7fbb4c003530 con 0x7fbb64156330 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 msgr2=0x7fbb640ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb640ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 >> v1:172.21.15.157:6789/0 conn(0x7fbb640cdee0 legacy=0x7fbb640b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6415f6e0 con 0x7fbb64156330 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fbb54001680 con 0x7fbb64156330 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 >> v1:172.21.15.136:6789/0 conn(0x7fbb64156330 legacy=0x7fbb6415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb4c0039f0 0x7fbb4c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb4c004d50 0x7fbb4c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbb641602f0 con 0x7fbb4c004d50 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c00bb90 con 0x7fbb4c0039f0 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb4c00bd90 con 0x7fbb640b5ba0 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb4c004d50 0x7fbb4c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb62ffd640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb4c0039f0 0x7fbb4c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb627fc640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb4c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 -- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 msgr2=0x7fbb4c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb640b5ba0 0x7fbb4c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 -- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb4c0039f0 msgr2=0x7fbb4c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb4c0039f0 0x7fbb4c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.544+0000 7fbb637fe640 1 -- 172.21.15.136:0/2464623474 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4c003530 con 0x7fbb4c004d50 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb637fe640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb4c004d50 0x7fbb4c005160 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbb640b53f0 tx=0x7fbb5c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd407dc1b5016f65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0b2070 con 0x7fbb4c004d50 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0a5c50 con 0x7fbb4c004d50 2024-04-25T11:37:07.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb61ffb640 1 -- 172.21.15.136:0/2464623474 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c009c10 con 0x7fbb4c004d50 2024-04-25T11:37:07.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 -- 172.21.15.136:0/2464623474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb4c004d50 msgr2=0x7fbb4c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 --2- 172.21.15.136:0/2464623474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb4c004d50 0x7fbb4c005160 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbb640b53f0 tx=0x7fbb5c01cc40 comp rx=0 tx=0).stop 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb637fe640 1 -- 172.21.15.136:0/2464623474 reap_dead start 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 -- 172.21.15.136:0/2464623474 shutdown_connections 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 -- 172.21.15.136:0/2464623474 >> 172.21.15.136:0/2464623474 conn(0x7fbb6406d6c0 msgr2=0x7fbb6406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 -- 172.21.15.136:0/2464623474 shutdown_connections 2024-04-25T11:37:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.548+0000 7fbb69321640 1 -- 172.21.15.136:0/2464623474 wait complete. 2024-04-25T11:37:07.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 Processor -- start 2024-04-25T11:37:07.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 -- start start 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 0x7fbb641e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6414f000 0x7fbb6414f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6415ff80 con 0x7fbb6414d4a0 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6415fca0 con 0x7fbb6415cfa0 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb69321640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb640cd320 con 0x7fbb6414f000 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38192/0 (socket says 172.21.15.136:38192) 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 -- 172.21.15.136:0/1920701080 learned_addr learned my addr 172.21.15.136:0/1920701080 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb637fe640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6414f000 0x7fbb6414f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb62ffd640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 0x7fbb641e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 -- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6414f000 msgr2=0x7fbb6414f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6414f000 0x7fbb6414f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 -- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 msgr2=0x7fbb641e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 0x7fbb641e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb627fc640 1 -- 172.21.15.136:0/1920701080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6414e560 con 0x7fbb6414d4a0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.552+0000 7fbb62ffd640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 0x7fbb641e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb627fc640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800e5b0 tx=0x7fbb5800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb580052e0 con 0x7fbb6414d4a0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb58018070 con 0x7fbb6414d4a0 2024-04-25T11:37:07.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6414da70 con 0x7fbb6414d4a0 2024-04-25T11:37:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58003820 con 0x7fbb6414d4a0 2024-04-25T11:37:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6414e800 con 0x7fbb6414d4a0 2024-04-25T11:37:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb6406a350 con 0x7fbb6414d4a0 2024-04-25T11:37:07.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbb580039c0 con 0x7fbb6414d4a0 2024-04-25T11:37:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 0x7fbb50071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fbb5808f270 con 0x7fbb6414d4a0 2024-04-25T11:37:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.556+0000 7fbb62ffd640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 0x7fbb50071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:07.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.560+0000 7fbb62ffd640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 0x7fbb50071940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001060 tx=0x7fbb540011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:07.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.560+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb5805bbb0 con 0x7fbb6414d4a0 2024-04-25T11:37:07.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.840+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb640cd9c0 con 0x7fbb6414d4a0 2024-04-25T11:37:07.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.840+0000 7fbb4b7fe640 1 -- 172.21.15.136:0/1920701080 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+6100 (secure 0 0 0) 0x7fbb58005480 con 0x7fbb6414d4a0 2024-04-25T11:37:07.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:07.824 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:03.988268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:resolve","state_seq":12,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:07.825 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 69 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 msgr2=0x7fbb50071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 0x7fbb50071940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001060 tx=0x7fbb540011a0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 msgr2=0x7fbb641e79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800e5b0 tx=0x7fbb5800ea70 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 shutdown_connections 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb5006f430 0x7fbb50071940 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6414f000 0x7fbb6414f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb6414d4a0 0x7fbb641e79b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 --2- 172.21.15.136:0/1920701080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6415cfa0 0x7fbb641e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 >> 172.21.15.136:0/1920701080 conn(0x7fbb6406d6c0 msgr2=0x7fbb6414c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 shutdown_connections 2024-04-25T11:37:07.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:07.844+0000 7fbb69321640 1 -- 172.21.15.136:0/1920701080 wait complete. 2024-04-25T11:37:07.846 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:09.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:10.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 Processor -- start 2024-04-25T11:37:10.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 -- start start 2024-04-25T11:37:10.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9cc00ca720 0x7f9cc00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc00cbad0 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc00c7c20 con 0x7f9cc00ce260 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc63a7640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc00c8550 con 0x7f9cc00ca720 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9cc00c8cd0 0x7f9cc014f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41144/0 (socket says 172.21.15.136:41144) 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbffff640 1 -- 172.21.15.136:0/2538950661 learned_addr learned my addr 172.21.15.136:0/2538950661 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cc4944640 1 --2- 172.21.15.136:0/2538950661 >> v2:172.21.15.157:3300/0 conn(0x7f9cc00ca720 0x7f9cc00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149983803 0 0) 0x7f9cc00cbad0 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cac003530 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 975410877 0 0) 0x7f9cac003530 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 >> v2:172.21.15.157:3300/0 conn(0x7f9cc00ca720 msgr2=0x7f9cc00ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.028+0000 7f9cbf7fe640 1 --2- 172.21.15.136:0/2538950661 >> v2:172.21.15.157:3300/0 conn(0x7f9cc00ca720 0x7f9cc00ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 >> v1:172.21.15.157:6789/0 conn(0x7f9cc00ce260 legacy=0x7f9cc00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc00c7f00 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f9cb0001680 con 0x7f9cc00c8cd0 2024-04-25T11:37:10.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 >> v1:172.21.15.136:6789/0 conn(0x7f9cc00c8cd0 legacy=0x7f9cc014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cac0039c0 0x7f9cac003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cac004810 0x7f9cac004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc00c8550 con 0x7f9cac005160 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac00c020 con 0x7f9cac004810 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cac00c220 con 0x7f9cac0039c0 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cb7fff640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cac004810 0x7f9cac004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbffff640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cac0039c0 0x7f9cac003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 -- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cac0039c0 msgr2=0x7f9cac003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cac0039c0 0x7f9cac003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 -- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cac004810 msgr2=0x7f9cac004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cac004810 0x7f9cac004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 -- 172.21.15.136:0/2538950661 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac003530 con 0x7f9cac005160 2024-04-25T11:37:10.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cb7fff640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cac004810 0x7f9cac004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 -- 172.21.15.136:0/2538950661 reap_dead start 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc4944640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cac00b9d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009fc0 tx=0x7f9cb801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498ac04bc2838a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8009c10 con 0x7f9cac005160 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb80a5d10 con 0x7f9cac005160 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cbf7fe640 1 -- 172.21.15.136:0/2538950661 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8003be0 con 0x7f9cac005160 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 -- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 msgr2=0x7f9cac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cac00b9d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009fc0 tx=0x7f9cb801ccc0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 -- 172.21.15.136:0/2538950661 shutdown_connections 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/2538950661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cac00b9d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 -- 172.21.15.136:0/2538950661 >> 172.21.15.136:0/2538950661 conn(0x7f9cc006f9d0 msgr2=0x7f9cc006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 -- 172.21.15.136:0/2538950661 shutdown_connections 2024-04-25T11:37:10.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.032+0000 7f9cc63a7640 1 -- 172.21.15.136:0/2538950661 wait complete. 2024-04-25T11:37:10.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 Processor -- start 2024-04-25T11:37:10.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 -- start start 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc014d480 0x7f9cc015c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc015ccf0 0x7f9cc014aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc00c91e0 con 0x7f9cac005160 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc00cbcd0 con 0x7f9cc014d480 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc63a7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc00c9360 con 0x7f9cc015ccf0 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38202/0 (socket says 172.21.15.136:38202) 2024-04-25T11:37:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 -- 172.21.15.136:0/4242099733 learned_addr learned my addr 172.21.15.136:0/4242099733 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cc4944640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc015ccf0 0x7f9cc014aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cb7fff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc014d480 0x7f9cc015c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 -- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc015ccf0 msgr2=0x7f9cc014aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc015ccf0 0x7f9cc014aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 -- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc014d480 msgr2=0x7f9cc015c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc014d480 0x7f9cc015c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.036+0000 7f9cbffff640 1 -- 172.21.15.136:0/4242099733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc014e9f0 con 0x7f9cac005160 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbffff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000e7b0 tx=0x7f9cb000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000c6c0 con 0x7f9cac005160 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb0018070 con 0x7f9cac005160 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000cac0 con 0x7f9cac005160 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc014ecd0 con 0x7f9cac005160 2024-04-25T11:37:10.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc014cf60 con 0x7f9cac005160 2024-04-25T11:37:10.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc006a350 con 0x7f9cac005160 2024-04-25T11:37:10.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9cb000cc60 con 0x7f9cac005160 2024-04-25T11:37:10.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 0x7f9ca0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:10.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f9cb008f530 con 0x7f9cac005160 2024-04-25T11:37:10.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cb7fff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 0x7f9ca0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:10.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.040+0000 7f9cb7fff640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 0x7f9ca0071940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9cc01531d0 tx=0x7f9ca8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:10.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.048+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cb005b7e0 con 0x7f9cac005160 2024-04-25T11:37:10.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.264+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc014cd10 con 0x7f9cac005160 2024-04-25T11:37:10.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.264+0000 7f9cbd7fa640 1 -- 172.21.15.136:0/4242099733 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+6102 (secure 0 0 0) 0x7f9cb005fae0 con 0x7f9cac005160 2024-04-25T11:37:10.249 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:10.249 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:09.321714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:37:10.249 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 70 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 msgr2=0x7f9ca0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 0x7f9ca0071940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9cc01531d0 tx=0x7f9ca8009040 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 msgr2=0x7f9cc0151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000e7b0 tx=0x7f9cb000ec70 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 shutdown_connections 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9ca006f430 0x7f9ca0071940 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc015ccf0 0x7f9cc014aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc014d480 0x7f9cc015c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 --2- 172.21.15.136:0/4242099733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cac005160 0x7f9cc0151e70 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 >> 172.21.15.136:0/4242099733 conn(0x7f9cc006f9d0 msgr2=0x7f9cc00c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 shutdown_connections 2024-04-25T11:37:10.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:10.268+0000 7f9cc63a7640 1 -- 172.21.15.136:0/4242099733 wait complete. 2024-04-25T11:37:10.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:12.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:12.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 Processor -- start 2024-04-25T11:37:12.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 -- start start 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c8414f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8c840ca720 0x7f8c840ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c840cbad0 con 0x7f8c840ce260 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8c840c8550 con 0x7f8c840ca720 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c89daf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c840c7c20 con 0x7f8c840c8cd0 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c82ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8c840ce260 0x7f8c840ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41156/0 (socket says 172.21.15.136:41156) 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c82ffd640 1 -- 172.21.15.136:0/2724394187 learned_addr learned my addr 172.21.15.136:0/2724394187 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c837fe640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c8414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c83fff640 1 --2- 172.21.15.136:0/2724394187 >> v2:172.21.15.157:3300/0 conn(0x7f8c840ca720 0x7f8c840ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1134491695 0 0) 0x7f8c840cbad0 con 0x7f8c840ce260 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c6c005740 con 0x7f8c840ce260 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971939838 0 0) 0x7f8c6c005740 con 0x7f8c840ce260 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 msgr2=0x7f8c8414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c8414f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 >> v2:172.21.15.157:3300/0 conn(0x7f8c840ca720 msgr2=0x7f8c840ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 --2- 172.21.15.136:0/2724394187 >> v2:172.21.15.157:3300/0 conn(0x7f8c840ca720 0x7f8c840ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c840c7f00 con 0x7f8c840ce260 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8c78001680 con 0x7f8c840ce260 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c837fe640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c8414f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 >> v1:172.21.15.136:6789/0 conn(0x7f8c840ce260 legacy=0x7f8c840ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c6c005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c6c006130 0x7f8c6c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c6c007370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840c8550 con 0x7f8c6c006f60 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840c7c20 con 0x7f8c6c006130 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c6c0035b0 con 0x7f8c840c8cd0 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c83fff640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c6c007370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c837fe640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c6c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.460+0000 7f8c82ffd640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c6c006130 0x7f8c6c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 -- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 msgr2=0x7f8c6c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c840c8cd0 0x7f8c6c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 -- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c6c006130 msgr2=0x7f8c6c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c6c006130 0x7f8c6c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 -- 172.21.15.136:0/2724394187 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6c005740 con 0x7f8c6c006f60 2024-04-25T11:37:12.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c82ffd640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c6c006130 0x7f8c6c006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:12.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c6c007370 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009f90 tx=0x7f8c7c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368b3ff90600b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0b2070 con 0x7f8c6c006f60 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7c0a5c90 con 0x7f8c6c006f60 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c827fc640 1 -- 172.21.15.136:0/2724394187 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c009c10 con 0x7f8c6c006f60 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 -- 172.21.15.136:0/2724394187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 msgr2=0x7f8c6c007370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 --2- 172.21.15.136:0/2724394187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c6c007370 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009f90 tx=0x7f8c7c01cc80 comp rx=0 tx=0).stop 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c83fff640 1 -- 172.21.15.136:0/2724394187 reap_dead start 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 -- 172.21.15.136:0/2724394187 shutdown_connections 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 -- 172.21.15.136:0/2724394187 >> 172.21.15.136:0/2724394187 conn(0x7f8c8406f9d0 msgr2=0x7f8c8406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 -- 172.21.15.136:0/2724394187 shutdown_connections 2024-04-25T11:37:12.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.464+0000 7f8c89daf640 1 -- 172.21.15.136:0/2724394187 wait complete. 2024-04-25T11:37:12.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 Processor -- start 2024-04-25T11:37:12.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 -- start start 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c84155b90 0x7f8c84148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 0x7f8c8415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840cb720 con 0x7f8c6c006f60 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840c91e0 con 0x7f8c84151d80 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c89daf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c841f1d70 con 0x7f8c84155b90 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38214/0 (socket says 172.21.15.136:38214) 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.468+0000 7f8c837fe640 1 -- 172.21.15.136:0/67562069 learned_addr learned my addr 172.21.15.136:0/67562069 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c82ffd640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c84155b90 0x7f8c84148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c83fff640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 0x7f8c8415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 -- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c84155b90 msgr2=0x7f8c84148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c84155b90 0x7f8c84148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 -- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 msgr2=0x7f8c8415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 0x7f8c8415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 -- 172.21.15.136:0/67562069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c841f18a0 con 0x7f8c6c006f60 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c83fff640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 0x7f8c8415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c837fe640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8c74011a00 tx=0x7f8c74011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7400fb50 con 0x7f8c6c006f60 2024-04-25T11:37:12.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7400fcf0 con 0x7f8c6c006f60 2024-04-25T11:37:12.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c84154e00 con 0x7f8c6c006f60 2024-04-25T11:37:12.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7400fea0 con 0x7f8c6c006f60 2024-04-25T11:37:12.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c84156830 con 0x7f8c6c006f60 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8406a350 con 0x7f8c6c006f60 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c74012040 con 0x7f8c6c006f60 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 0x7f8c70071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f8c7408fd80 con 0x7f8c6c006f60 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.472+0000 7f8c82ffd640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 0x7f8c70071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:12.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.476+0000 7f8c82ffd640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 0x7f8c70071910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ce0b0 tx=0x7f8c78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:12.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.480+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8c74019070 con 0x7f8c6c006f60 2024-04-25T11:37:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c841f0f80 con 0x7f8c6c006f60 2024-04-25T11:37:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c6bfff640 1 -- 172.21.15.136:0/67562069 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+6102 (secure 0 0 0) 0x7f8c7405c680 con 0x7f8c6c006f60 2024-04-25T11:37:12.702 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:12.703 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:09.321714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517]},"id":1}]} 2024-04-25T11:37:12.703 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 70 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 msgr2=0x7f8c70071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 0x7f8c70071910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ce0b0 tx=0x7f8c78009040 comp rx=0 tx=0).stop 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 msgr2=0x7f8c841f1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8c74011a00 tx=0x7f8c74011ec0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 shutdown_connections 2024-04-25T11:37:12.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c84151d80 0x7f8c8415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c7006f400 0x7f8c70071910 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c84155b90 0x7f8c84148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 --2- 172.21.15.136:0/67562069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c006f60 0x7f8c841f1360 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 >> 172.21.15.136:0/67562069 conn(0x7f8c8406f9d0 msgr2=0x7f8c8406e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 shutdown_connections 2024-04-25T11:37:12.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:12.720+0000 7f8c89daf640 1 -- 172.21.15.136:0/67562069 wait complete. 2024-04-25T11:37:12.727 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5356": { "addr": "172.21.15.157:6836/3463942074", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3463942074, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3463942074, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5356, "incarnation": 50, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 29 }, "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:37:09.321714+0000", "qdb_cluster": [ 5356, 5385, 5449, 5517 ], "qdb_leader": 5356, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5356, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:37:14.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:14.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 Processor -- start 2024-04-25T11:37:14.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 -- start start 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f48040b5060 0x7f48040c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f480415ab40 0x7f48040ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480415d750 con 0x7f48040ca290 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f480415da30 con 0x7f48040b5060 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f480b3ba640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f480415dd10 con 0x7f480415ab40 2024-04-25T11:37:14.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f48040b5060 0x7f48040c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f48040b5060 0x7f48040c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59484/0 (socket says 172.21.15.136:59484) 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 -- 172.21.15.136:0/3594164726 learned_addr learned my addr 172.21.15.136:0/3594164726 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809957640 1 --2- 172.21.15.136:0/3594164726 >> v2:172.21.15.157:3300/0 conn(0x7f480415ab40 0x7f48040ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 -- 172.21.15.136:0/3594164726 >> v2:172.21.15.157:3300/0 conn(0x7f480415ab40 msgr2=0x7f48040ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 --2- 172.21.15.136:0/3594164726 >> v2:172.21.15.157:3300/0 conn(0x7f480415ab40 0x7f48040ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 -- 172.21.15.136:0/3594164726 >> v1:172.21.15.157:6789/0 conn(0x7f48040ca290 legacy=0x7f480415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.904+0000 7f4809156640 1 -- 172.21.15.136:0/3594164726 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480415dff0 con 0x7f48040b5060 2024-04-25T11:37:14.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f4809156640 1 --2- 172.21.15.136:0/3594164726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040b5060 0x7f48040c4520 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f48040b4fe0 tx=0x7f480001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4827685b0603f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:14.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f47fbfff640 1 -- 172.21.15.136:0/3594164726 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000b5070 con 0x7f48040b5060 2024-04-25T11:37:14.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f47fbfff640 1 -- 172.21.15.136:0/3594164726 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48000a8a10 con 0x7f48040b5060 2024-04-25T11:37:14.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f47fbfff640 1 -- 172.21.15.136:0/3594164726 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000a8de0 con 0x7f48040b5060 2024-04-25T11:37:14.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 -- 172.21.15.136:0/3594164726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040b5060 msgr2=0x7f48040c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:14.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 --2- 172.21.15.136:0/3594164726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040b5060 0x7f48040c4520 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f48040b4fe0 tx=0x7f480001ed10 comp rx=0 tx=0).stop 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 -- 172.21.15.136:0/3594164726 shutdown_connections 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 --2- 172.21.15.136:0/3594164726 >> v2:172.21.15.157:3300/0 conn(0x7f480415ab40 0x7f48040ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 --2- 172.21.15.136:0/3594164726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040b5060 0x7f48040c4520 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 -- 172.21.15.136:0/3594164726 >> 172.21.15.136:0/3594164726 conn(0x7f4804070f70 msgr2=0x7f4804068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 -- 172.21.15.136:0/3594164726 shutdown_connections 2024-04-25T11:37:14.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.908+0000 7f480b3ba640 1 -- 172.21.15.136:0/3594164726 wait complete. 2024-04-25T11:37:14.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 Processor -- start 2024-04-25T11:37:14.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 -- start start 2024-04-25T11:37:14.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480415ab40 0x7f480414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4804150a30 0x7f480415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f480415d3f0 con 0x7f48040ca290 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4804159f10 con 0x7f4804150a30 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f480b3ba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f48040c4a60 con 0x7f480415ab40 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f4809156640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f4809156640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59488/0 (socket says 172.21.15.136:59488) 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.912+0000 7f4809156640 1 -- 172.21.15.136:0/3745677886 learned_addr learned my addr 172.21.15.136:0/3745677886 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4808955640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480415ab40 0x7f480414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809957640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4804150a30 0x7f480415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 -- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480415ab40 msgr2=0x7f480414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480415ab40 0x7f480414faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 -- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4804150a30 msgr2=0x7f480415c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4804150a30 0x7f480415c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 -- 172.21.15.136:0/3745677886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48041f10d0 con 0x7f48040ca290 2024-04-25T11:37:14.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4809156640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f48000b3040 tx=0x7f480001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000b5070 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48000b0b60 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000bb410 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f480b3ba640 1 -- 172.21.15.136:0/3745677886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480415afc0 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f480b3ba640 1 -- 172.21.15.136:0/3745677886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f480415b560 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e00052a0 con 0x7f48040ca290 2024-04-25T11:37:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f48000c2060 con 0x7f48040ca290 2024-04-25T11:37:14.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 0x7f47e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:14.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f480012ca90 con 0x7f48040ca290 2024-04-25T11:37:14.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.916+0000 7f4808955640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 0x7f47e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:14.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.920+0000 7f4808955640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 0x7f47e8071910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4804159ee0 tx=0x7f47f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:14.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:14.924+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4800018870 con 0x7f48040ca290 2024-04-25T11:37:15.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47e0005860 con 0x7f48040ca290 2024-04-25T11:37:15.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47f9ffb640 1 -- 172.21.15.136:0/3745677886 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6099 (secure 0 0 0) 0x7f48000f8cb0 con 0x7f48040ca290 2024-04-25T11:37:15.143 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:15.143 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:14.354859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:15.144 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 71 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 msgr2=0x7f47e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 0x7f47e8071910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4804159ee0 tx=0x7f47f4009040 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 msgr2=0x7f48041f0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.160+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f48000b3040 tx=0x7f480001bab0 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 shutdown_connections 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4804150a30 0x7f480415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f47e806f400 0x7f47e8071910 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f480415ab40 0x7f480414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 --2- 172.21.15.136:0/3745677886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48040ca290 0x7f48041f0b90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 >> 172.21.15.136:0/3745677886 conn(0x7f4804070f70 msgr2=0x7f48040cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:15.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 shutdown_connections 2024-04-25T11:37:15.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:15.164+0000 7f47e77fe640 1 -- 172.21.15.136:0/3745677886 wait complete. 2024-04-25T11:37:15.166 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:17.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:17.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 Processor -- start 2024-04-25T11:37:17.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 -- start start 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89740cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f897415ff60 0x7f89740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897415c6f0 con 0x7f8974160370 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f89740c6310 con 0x7f897415ff60 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.348+0000 7f897aee8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f897415c990 con 0x7f897415c0d0 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8974160370 0x7f8974071590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43786/0 (socket says 172.21.15.136:43786) 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 -- 172.21.15.136:0/3226092714 learned_addr learned my addr 172.21.15.136:0/3226092714 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8978c84640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89740cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8973fff640 1 --2- 172.21.15.136:0/3226092714 >> v2:172.21.15.157:3300/0 conn(0x7f897415ff60 0x7f89740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1871082294 0 0) 0x7f897415c6f0 con 0x7f8974160370 2024-04-25T11:37:17.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8964003530 con 0x7f8974160370 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2871415364 0 0) 0x7f8964003530 con 0x7f8974160370 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 msgr2=0x7f89740cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89740cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 >> v2:172.21.15.157:3300/0 conn(0x7f897415ff60 msgr2=0x7f89740ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 --2- 172.21.15.136:0/3226092714 >> v2:172.21.15.157:3300/0 conn(0x7f897415ff60 0x7f89740ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897415cc30 con 0x7f8974160370 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f896c001a00 con 0x7f8974160370 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 >> v1:172.21.15.136:6789/0 conn(0x7f8974160370 legacy=0x7f8974071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89640039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8964003f40 0x7f8964004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8964004d50 0x7f8964005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f89740c6310 con 0x7f8964004d50 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f897415c990 con 0x7f8964003f40 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f896400bab0 con 0x7f897415c0d0 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8964004d50 0x7f8964005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8973fff640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8964003f40 0x7f8964004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8978c84640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89640039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 -- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 msgr2=0x7f89640039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f897415c0d0 0x7f89640039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 -- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8964003f40 msgr2=0x7f8964004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:17.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8964003f40 0x7f8964004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 -- 172.21.15.136:0/3226092714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8964003530 con 0x7f8964004d50 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f8979485640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8964004d50 0x7f8964005160 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f89740cb3c0 tx=0x7f896c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a53337e6f6863c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c00a4c0 con 0x7f8964004d50 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896c0ad650 con 0x7f8964004d50 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.352+0000 7f89737fe640 1 -- 172.21.15.136:0/3226092714 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c0ada80 con 0x7f8964004d50 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- 172.21.15.136:0/3226092714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8964004d50 msgr2=0x7f8964005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 --2- 172.21.15.136:0/3226092714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8964004d50 0x7f8964005160 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f89740cb3c0 tx=0x7f896c01d360 comp rx=0 tx=0).stop 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f8979485640 1 -- 172.21.15.136:0/3226092714 reap_dead start 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- 172.21.15.136:0/3226092714 shutdown_connections 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- 172.21.15.136:0/3226092714 >> 172.21.15.136:0/3226092714 conn(0x7f897406f9d0 msgr2=0x7f897406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- 172.21.15.136:0/3226092714 shutdown_connections 2024-04-25T11:37:17.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- 172.21.15.136:0/3226092714 wait complete. 2024-04-25T11:37:17.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 Processor -- start 2024-04-25T11:37:17.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.356+0000 7f897aee8640 1 -- start start 2024-04-25T11:37:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8974144830 0x7f8974144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89741f2580 0x7f89741f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f897415cf10 con 0x7f89740c8110 2024-04-25T11:37:17.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f897415fa40 con 0x7f89741f2580 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f89740c9170 con 0x7f8974144830 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59502/0 (socket says 172.21.15.136:59502) 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 -- 172.21.15.136:0/652366724 learned_addr learned my addr 172.21.15.136:0/652366724 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8973fff640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8974144830 0x7f8974144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8979485640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89741f2580 0x7f89741f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 -- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8974144830 msgr2=0x7f8974144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8974144830 0x7f8974144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 -- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89741f2580 msgr2=0x7f89741f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89741f2580 0x7f89741f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 -- 172.21.15.136:0/652366724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89741f5560 con 0x7f89740c8110 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f8978c84640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8968011610 tx=0x7f8968011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896800f8f0 con 0x7f89740c8110 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896800fa90 con 0x7f89740c8110 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89741f5800 con 0x7f89740c8110 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896800fe40 con 0x7f89740c8110 2024-04-25T11:37:17.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89741f5da0 con 0x7f89740c8110 2024-04-25T11:37:17.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897406a350 con 0x7f89740c8110 2024-04-25T11:37:17.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8968015460 con 0x7f89740c8110 2024-04-25T11:37:17.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.360+0000 7f89717fa640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 0x7f8958071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:17.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.364+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f8968090300 con 0x7f89740c8110 2024-04-25T11:37:17.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.364+0000 7f8973fff640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 0x7f8958071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:17.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.364+0000 7f8973fff640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 0x7f8958071940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f897414ad70 tx=0x7f8960008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:17.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.368+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8968019030 con 0x7f89740c8110 2024-04-25T11:37:17.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.596+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8974145180 con 0x7f89740c8110 2024-04-25T11:37:17.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.596+0000 7f89717fa640 1 -- 172.21.15.136:0/652366724 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6099 (secure 0 0 0) 0x7f896805cb70 con 0x7f89740c8110 2024-04-25T11:37:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:17.580 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:14.354859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:17.580 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 71 2024-04-25T11:37:17.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 msgr2=0x7f8958071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 0x7f8958071940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f897414ad70 tx=0x7f8960008040 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 msgr2=0x7f897406c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8968011610 tx=0x7f8968011ad0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 shutdown_connections 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89741f2580 0x7f89741f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895806f430 0x7f8958071940 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8974144830 0x7f8974144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 --2- 172.21.15.136:0/652366724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89740c8110 0x7f897406c840 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 >> 172.21.15.136:0/652366724 conn(0x7f897406f9d0 msgr2=0x7f89740c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 shutdown_connections 2024-04-25T11:37:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:17.600+0000 7f897aee8640 1 -- 172.21.15.136:0/652366724 wait complete. 2024-04-25T11:37:17.601 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:19.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:19.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.804+0000 7ffa8b66e640 1 Processor -- start 2024-04-25T11:37:19.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.804+0000 7ffa8b66e640 1 -- start start 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8b66e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffa8414f840 0x7ffa84074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8b66e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa840c3750 con 0x7ffa84073200 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8b66e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa840c3190 con 0x7ffa840c4340 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8b66e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffa8406dc00 con 0x7ffa8414f840 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa88c09640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ffa84073200 0x7ffa840c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43802/0 (socket says 172.21.15.136:43802) 2024-04-25T11:37:19.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa88c09640 1 -- 172.21.15.136:0/1982970030 learned_addr learned my addr 172.21.15.136:0/1982970030 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:37:19.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8940a640 1 --2- 172.21.15.136:0/1982970030 >> v2:172.21.15.157:3300/0 conn(0x7ffa8414f840 0x7ffa84074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4168150116 0 0) 0x7ffa840c3750 con 0x7ffa84073200 2024-04-25T11:37:19.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa70003530 con 0x7ffa84073200 2024-04-25T11:37:19.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1776326259 0 0) 0x7ffa70003530 con 0x7ffa84073200 2024-04-25T11:37:19.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 >> v2:172.21.15.157:3300/0 conn(0x7ffa8414f840 msgr2=0x7ffa84074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 --2- 172.21.15.136:0/1982970030 >> v2:172.21.15.157:3300/0 conn(0x7ffa8414f840 0x7ffa84074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 >> v1:172.21.15.157:6789/0 conn(0x7ffa840c4340 legacy=0x7ffa841636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa840c3470 con 0x7ffa84073200 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7ffa78001680 con 0x7ffa84073200 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 >> v1:172.21.15.136:6789/0 conn(0x7ffa84073200 legacy=0x7ffa840c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa700039c0 0x7ffa70003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa70004810 0x7ffa70004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa7000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8406dc00 con 0x7ffa700051d0 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7000c170 con 0x7ffa70004810 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7000c370 con 0x7ffa700039c0 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa7000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa8940a640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa700039c0 0x7ffa70003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa88c09640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa70004810 0x7ffa70004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 -- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa700039c0 msgr2=0x7ffa70003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa700039c0 0x7ffa70003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 -- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa70004810 msgr2=0x7ffa70004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa70004810 0x7ffa70004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.808+0000 7ffa89c0b640 1 -- 172.21.15.136:0/1982970030 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa70003530 con 0x7ffa700051d0 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa89c0b640 1 -- 172.21.15.136:0/1982970030 reap_dead start 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa89c0b640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa7000b9d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c001e40 tx=0x7ffa7c01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54a8234082d47e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c001a90 con 0x7ffa700051d0 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa7c003b80 con 0x7ffa700051d0 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa77fff640 1 -- 172.21.15.136:0/1982970030 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c0050d0 con 0x7ffa700051d0 2024-04-25T11:37:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 -- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 msgr2=0x7ffa7000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa7000b9d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c001e40 tx=0x7ffa7c01c6a0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 -- 172.21.15.136:0/1982970030 shutdown_connections 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/1982970030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa7000b9d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 -- 172.21.15.136:0/1982970030 >> 172.21.15.136:0/1982970030 conn(0x7ffa840ca5a0 msgr2=0x7ffa840ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 -- 172.21.15.136:0/1982970030 shutdown_connections 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.812+0000 7ffa8b66e640 1 -- 172.21.15.136:0/1982970030 wait complete. 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 Processor -- start 2024-04-25T11:37:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 -- start start 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa841588d0 0x7ffa8415c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa840cc8a0 0x7ffa84148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa84153380 con 0x7ffa700051d0 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8414fc50 con 0x7ffa840cc8a0 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8b66e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa84073610 con 0x7ffa841588d0 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59512/0 (socket says 172.21.15.136:59512) 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 -- 172.21.15.136:0/283483851 learned_addr learned my addr 172.21.15.136:0/283483851 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:19.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa89c0b640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa840cc8a0 0x7ffa84148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa88c09640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa841588d0 0x7ffa8415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 -- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa841588d0 msgr2=0x7ffa8415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa841588d0 0x7ffa8415c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 -- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa840cc8a0 msgr2=0x7ffa84148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa840cc8a0 0x7ffa84148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 -- 172.21.15.136:0/283483851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa841f35f0 con 0x7ffa700051d0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.816+0000 7ffa8940a640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011590 tx=0x7ffa80011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80015930 con 0x7ffa700051d0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa80015ad0 con 0x7ffa700051d0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa841f38d0 con 0x7ffa700051d0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000f460 con 0x7ffa700051d0 2024-04-25T11:37:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa841f3e90 con 0x7ffa700051d0 2024-04-25T11:37:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa8406a350 con 0x7ffa700051d0 2024-04-25T11:37:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffa8000f600 con 0x7ffa700051d0 2024-04-25T11:37:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 0x7ffa68071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7ffa80090250 con 0x7ffa700051d0 2024-04-25T11:37:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa88c09640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 0x7ffa68071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:19.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.820+0000 7ffa88c09640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 0x7ffa68071940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffa841edc10 tx=0x7ffa78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:19.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:19.824+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffa80019030 con 0x7ffa700051d0 2024-04-25T11:37:20.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.056+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa840c2ed0 con 0x7ffa700051d0 2024-04-25T11:37:20.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa75ffb640 1 -- 172.21.15.136:0/283483851 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6099 (secure 0 0 0) 0x7ffa8005cac0 con 0x7ffa700051d0 2024-04-25T11:37:20.042 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:20.042 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:14.354859+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:20.043 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 71 2024-04-25T11:37:20.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 msgr2=0x7ffa68071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:20.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 0x7ffa68071940 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffa841edc10 tx=0x7ffa78009040 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 msgr2=0x7ffa84149360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011590 tx=0x7ffa80011a50 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 shutdown_connections 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa840cc8a0 0x7ffa84148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ffa6806f430 0x7ffa68071940 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa841588d0 0x7ffa8415c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 --2- 172.21.15.136:0/283483851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa700051d0 0x7ffa84149360 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 >> 172.21.15.136:0/283483851 conn(0x7ffa840ca5a0 msgr2=0x7ffa8414ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 shutdown_connections 2024-04-25T11:37:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:20.060+0000 7ffa8b66e640 1 -- 172.21.15.136:0/283483851 wait complete. 2024-04-25T11:37:20.064 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:37:22.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:22.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 Processor -- start 2024-04-25T11:37:22.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 -- start start 2024-04-25T11:37:22.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdfa80ca290 0x7fdfa80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdfa815a430 0x7fdfa815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfa815d750 con 0x7fdfa80b5060 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdfa815da30 con 0x7fdfa815a430 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfad269640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdfa815dd10 con 0x7fdfa80ca290 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdfa815a430 0x7fdfa815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdfa815a430 0x7fdfa815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59520/0 (socket says 172.21.15.136:59520) 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 -- 172.21.15.136:0/3994396211 learned_addr learned my addr 172.21.15.136:0/3994396211 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa67fc640 1 --2- 172.21.15.136:0/3994396211 >> v2:172.21.15.157:3300/0 conn(0x7fdfa80ca290 0x7fdfa80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 -- 172.21.15.136:0/3994396211 >> v2:172.21.15.157:3300/0 conn(0x7fdfa80ca290 msgr2=0x7fdfa80ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 --2- 172.21.15.136:0/3994396211 >> v2:172.21.15.157:3300/0 conn(0x7fdfa80ca290 0x7fdfa80ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 -- 172.21.15.136:0/3994396211 >> v1:172.21.15.157:6789/0 conn(0x7fdfa80b5060 legacy=0x7fdfa80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.256+0000 7fdfa77fe640 1 -- 172.21.15.136:0/3994396211 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa815dff0 con 0x7fdfa815a430 2024-04-25T11:37:22.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfa77fe640 1 --2- 172.21.15.136:0/3994396211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa815a430 0x7fdfa815a840 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80d00d0 tx=0x7fdfa001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aebb2152444410c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:22.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfa5ffb640 1 -- 172.21.15.136:0/3994396211 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa00b4070 con 0x7fdfa815a430 2024-04-25T11:37:22.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfa5ffb640 1 -- 172.21.15.136:0/3994396211 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa00af400 con 0x7fdfa815a430 2024-04-25T11:37:22.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfa5ffb640 1 -- 172.21.15.136:0/3994396211 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa00af7d0 con 0x7fdfa815a430 2024-04-25T11:37:22.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 -- 172.21.15.136:0/3994396211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa815a430 msgr2=0x7fdfa815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 --2- 172.21.15.136:0/3994396211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa815a430 0x7fdfa815a840 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80d00d0 tx=0x7fdfa001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 -- 172.21.15.136:0/3994396211 shutdown_connections 2024-04-25T11:37:22.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 --2- 172.21.15.136:0/3994396211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa815a430 0x7fdfa815a840 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 --2- 172.21.15.136:0/3994396211 >> v2:172.21.15.157:3300/0 conn(0x7fdfa80ca290 0x7fdfa80ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.260+0000 7fdfad269640 1 -- 172.21.15.136:0/3994396211 >> 172.21.15.136:0/3994396211 conn(0x7fdfa8070f70 msgr2=0x7fdfa8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:22.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- 172.21.15.136:0/3994396211 shutdown_connections 2024-04-25T11:37:22.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- 172.21.15.136:0/3994396211 wait complete. 2024-04-25T11:37:22.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 Processor -- start 2024-04-25T11:37:22.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- start start 2024-04-25T11:37:22.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa80b5060 0x7fdfa814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfa815a430 0x7fdfa81480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa815ad80 con 0x7fdfa80ca290 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa8159f10 con 0x7fdfa80b5060 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfad269640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa815d3f0 con 0x7fdfa815a430 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfa67fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfa67fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59534/0 (socket says 172.21.15.136:59534) 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfa67fc640 1 -- 172.21.15.136:0/3598358781 learned_addr learned my addr 172.21.15.136:0/3598358781 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.264+0000 7fdfa6ffd640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa80b5060 0x7fdfa814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa77fe640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfa815a430 0x7fdfa81480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 -- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfa815a430 msgr2=0x7fdfa81480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfa815a430 0x7fdfa81480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 -- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa80b5060 msgr2=0x7fdfa814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa80b5060 0x7fdfa814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 -- 172.21.15.136:0/3598358781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa8151bd0 con 0x7fdfa80ca290 2024-04-25T11:37:22.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa67fc640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00e8a0 tx=0x7fdf9c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c017070 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c00c940 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa815c120 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c00ccf0 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa81f4800 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa8151ef0 con 0x7fdfa80ca290 2024-04-25T11:37:22.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdf9c01a070 con 0x7fdfa80ca290 2024-04-25T11:37:22.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 0x7fdf94071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:22.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fdf9c08fb70 con 0x7fdfa80ca290 2024-04-25T11:37:22.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.268+0000 7fdfa6ffd640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 0x7fdf94071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:22.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.272+0000 7fdfa6ffd640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 0x7fdf94071910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf98001770 tx=0x7fdf98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:22.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.276+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdf9c05bd90 con 0x7fdfa80ca290 2024-04-25T11:37:22.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.500+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfa806a350 con 0x7fdfa80ca290 2024-04-25T11:37:22.484 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.500+0000 7fdf8f7fe640 1 -- 172.21.15.136:0/3598358781 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6104 (secure 0 0 0) 0x7fdf9c060090 con 0x7fdfa80ca290 2024-04-25T11:37:22.484 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:22.484 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:37:22.485 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 72 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 msgr2=0x7fdf94071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 0x7fdf94071910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdf98001770 tx=0x7fdf98009040 comp rx=0 tx=0).stop 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 msgr2=0x7fdfa8147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00e8a0 tx=0x7fdf9c00eca0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 shutdown_connections 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdf9406f400 0x7fdf94071910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdfa815a430 0x7fdfa81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa80ca290 0x7fdfa8147ba0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 --2- 172.21.15.136:0/3598358781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdfa80b5060 0x7fdfa814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:22.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 >> 172.21.15.136:0/3598358781 conn(0x7fdfa8070f70 msgr2=0x7fdfa80ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:22.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 shutdown_connections 2024-04-25T11:37:22.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:22.504+0000 7fdfad269640 1 -- 172.21.15.136:0/3598358781 wait complete. 2024-04-25T11:37:22.508 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=4, replacing gid=5249 2024-04-25T11:37:22.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2024-04-25T11:37:24.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:37:24.041 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:37:25.181 INFO:tasks.ceph.mds.d.smithi157.stderr:2024-04-25T11:37:25.195+0000 7ffa711bc640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:37:28.509 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-25T11:37:28.509 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T11:37:28.509 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T11:37:28.512 INFO:tasks.ceph.mds.b:Started 2024-04-25T11:37:28.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:28.618 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:37:28.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 Processor -- start 2024-04-25T11:37:28.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 -- start start 2024-04-25T11:37:28.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f89700b5060 0x7f89700c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f897015ab40 0x7f89700ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897015d750 con 0x7f89700ca290 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f897015da30 con 0x7f897015ab40 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8978126640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f897015dd10 con 0x7f89700b5060 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f897015ab40 0x7f89700ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f897015ab40 0x7f89700ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51566/0 (socket says 172.21.15.136:51566) 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 -- 172.21.15.136:0/4010101778 learned_addr learned my addr 172.21.15.136:0/4010101778 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f8975ec2640 1 --2- 172.21.15.136:0/4010101778 >> v2:172.21.15.157:3300/0 conn(0x7f89700b5060 0x7f89700c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 -- 172.21.15.136:0/4010101778 >> v2:172.21.15.157:3300/0 conn(0x7f89700b5060 msgr2=0x7f89700c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 --2- 172.21.15.136:0/4010101778 >> v2:172.21.15.157:3300/0 conn(0x7f89700b5060 0x7f89700c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 -- 172.21.15.136:0/4010101778 >> v1:172.21.15.157:6789/0 conn(0x7f89700ca290 legacy=0x7f897015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.696+0000 7f89766c3640 1 -- 172.21.15.136:0/4010101778 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897015dff0 con 0x7f897015ab40 2024-04-25T11:37:28.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f89766c3640 1 --2- 172.21.15.136:0/4010101778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f897015ab40 0x7f89700ca6a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8970068ff0 tx=0x7f896801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1359cd4c8c055d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8974ec0640 1 -- 172.21.15.136:0/4010101778 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680b4070 con 0x7f897015ab40 2024-04-25T11:37:28.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8974ec0640 1 -- 172.21.15.136:0/4010101778 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89680af400 con 0x7f897015ab40 2024-04-25T11:37:28.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8974ec0640 1 -- 172.21.15.136:0/4010101778 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680af7d0 con 0x7f897015ab40 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 -- 172.21.15.136:0/4010101778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f897015ab40 msgr2=0x7f89700ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 --2- 172.21.15.136:0/4010101778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f897015ab40 0x7f89700ca6a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8970068ff0 tx=0x7f896801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 -- 172.21.15.136:0/4010101778 shutdown_connections 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 --2- 172.21.15.136:0/4010101778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f897015ab40 0x7f89700ca6a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 --2- 172.21.15.136:0/4010101778 >> v2:172.21.15.157:3300/0 conn(0x7f89700b5060 0x7f89700c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 -- 172.21.15.136:0/4010101778 >> 172.21.15.136:0/4010101778 conn(0x7f8970070f70 msgr2=0x7f8970068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 -- 172.21.15.136:0/4010101778 shutdown_connections 2024-04-25T11:37:28.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.700+0000 7f8978126640 1 -- 172.21.15.136:0/4010101778 wait complete. 2024-04-25T11:37:28.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 Processor -- start 2024-04-25T11:37:28.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 -- start start 2024-04-25T11:37:28.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89700ca290 0x7f8970146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 0x7f8970146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f897015d3f0 con 0x7f89700b5060 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8970159f10 con 0x7f897015ab40 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f89700c4a60 con 0x7f89700ca290 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51578/0 (socket says 172.21.15.136:51578) 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 -- 172.21.15.136:0/2100548934 learned_addr learned my addr 172.21.15.136:0/2100548934 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f89756c1640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89700ca290 0x7f8970146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f89766c3640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 0x7f8970146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 -- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89700ca290 msgr2=0x7f8970146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89700ca290 0x7f8970146860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 -- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 msgr2=0x7f8970146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 0x7f8970146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 -- 172.21.15.136:0/2100548934 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897014ff80 con 0x7f89700b5060 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f89766c3640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 0x7f8970146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8975ec2640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f896c00e8a0 tx=0x7f896c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c017070 con 0x7f89700b5060 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f896c00c940 con 0x7f89700b5060 2024-04-25T11:37:28.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.704+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89701f49c0 con 0x7f89700b5060 2024-04-25T11:37:28.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c00ccf0 con 0x7f89700b5060 2024-04-25T11:37:28.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f897014fca0 con 0x7f89700b5060 2024-04-25T11:37:28.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f896c01a070 con 0x7f89700b5060 2024-04-25T11:37:28.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f897006a350 con 0x7f89700b5060 2024-04-25T11:37:28.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89667fc640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 0x7f8954071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:28.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f896c08f0c0 con 0x7f89700b5060 2024-04-25T11:37:28.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89756c1640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 0x7f8954071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:28.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.708+0000 7f89756c1640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 0x7f8954071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f89700c4590 tx=0x7f8960009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:28.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.716+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f896c096070 con 0x7f89700b5060 2024-04-25T11:37:28.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.952+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89700cadd0 con 0x7f89700b5060 2024-04-25T11:37:28.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.952+0000 7f89667fc640 1 -- 172.21.15.136:0/2100548934 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6104 (secure 0 0 0) 0x7f896c05b1c0 con 0x7f89700b5060 2024-04-25T11:37:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:28.936 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[5356,5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:37:28.937 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 72 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 msgr2=0x7f8954071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 0x7f8954071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f89700c4590 tx=0x7f8960009040 comp rx=0 tx=0).stop 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 msgr2=0x7f89701f4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f896c00e8a0 tx=0x7f896c00eca0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 shutdown_connections 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f897015ab40 0x7f8970146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f895406f400 0x7f8954071910 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89700ca290 0x7f8970146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 --2- 172.21.15.136:0/2100548934 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89700b5060 0x7f89701f4480 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:28.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 >> 172.21.15.136:0/2100548934 conn(0x7f8970070f70 msgr2=0x7f897014a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:28.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 shutdown_connections 2024-04-25T11:37:28.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:28.956+0000 7f8978126640 1 -- 172.21.15.136:0/2100548934 wait complete. 2024-04-25T11:37:28.959 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T11:37:30.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:37:31.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.144+0000 7f6d0f367640 1 Processor -- start 2024-04-25T11:37:31.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.144+0000 7f6d0f367640 1 -- start start 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0f367640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0f367640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6d0815aae0 0x7f6d080c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0f367640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d080c9af0 con 0x7f6d080c79e0 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0f367640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6d0806cc10 con 0x7f6d0815aae0 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0f367640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6d080cb220 con 0x7f6d080c96e0 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d103640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6d0815aae0 0x7f6d080c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d103640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58900/0 (socket says 172.21.15.136:58900) 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6d0815aae0 0x7f6d080c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37238/0 (socket says 172.21.15.136:37238) 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0c902640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f6d080c79e0 0x7f6d0815a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:37920/0 (socket says 172.21.15.136:37920) 2024-04-25T11:37:31.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d103640 1 -- 172.21.15.136:0/2294333961 learned_addr learned my addr 172.21.15.136:0/2294333961 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:31.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 -- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 msgr2=0x7f6d081540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:37:31.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 -- 172.21.15.136:0/2294333961 >> v1:172.21.15.157:6789/0 conn(0x7f6d080c79e0 legacy=0x7f6d0815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d904640 1 -- 172.21.15.136:0/2294333961 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d080cb500 con 0x7f6d0815aae0 2024-04-25T11:37:31.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.148+0000 7f6d0d103640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:37:31.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0d904640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c7df0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009d90 tx=0x7f6d0001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7caca66fe3153481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:31.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6cfffff640 1 -- 172.21.15.136:0/2294333961 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000b4070 con 0x7f6d0815aae0 2024-04-25T11:37:31.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6cfffff640 1 -- 172.21.15.136:0/2294333961 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d000af600 con 0x7f6d0815aae0 2024-04-25T11:37:31.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6cfffff640 1 -- 172.21.15.136:0/2294333961 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000af9d0 con 0x7f6d0815aae0 2024-04-25T11:37:31.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 -- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 msgr2=0x7f6d080c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c7df0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009d90 tx=0x7f6d0001f150 comp rx=0 tx=0).stop 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 -- 172.21.15.136:0/2294333961 shutdown_connections 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c7df0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2294333961 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c96e0 0x7f6d081540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 -- 172.21.15.136:0/2294333961 >> 172.21.15.136:0/2294333961 conn(0x7f6d0806f9d0 msgr2=0x7f6d0806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 -- 172.21.15.136:0/2294333961 shutdown_connections 2024-04-25T11:37:31.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.152+0000 7f6d0f367640 1 -- 172.21.15.136:0/2294333961 wait complete. 2024-04-25T11:37:31.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 Processor -- start 2024-04-25T11:37:31.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 -- start start 2024-04-25T11:37:31.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c79e0 0x7f6d080cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6d080cf0f0 con 0x7f6d080c96e0 2024-04-25T11:37:31.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08154610 con 0x7f6d0815aae0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0f367640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6d080ced40 con 0x7f6d080c79e0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51582/0 (socket says 172.21.15.136:51582) 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 -- 172.21.15.136:0/2771928664 learned_addr learned my addr 172.21.15.136:0/2771928664 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0d904640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0d103640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c79e0 0x7f6d080cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 -- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c79e0 msgr2=0x7f6d080cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c79e0 0x7f6d080cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 -- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 msgr2=0x7f6d080c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0c902640 1 -- 172.21.15.136:0/2771928664 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d080cd3b0 con 0x7f6d080c96e0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.156+0000 7f6d0d904640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0c902640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800c370 tx=0x7f6cf800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80074e0 con 0x7f6d080c96e0 2024-04-25T11:37:31.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf8007680 con 0x7f6d080c96e0 2024-04-25T11:37:31.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d081f3520 con 0x7f6d080c96e0 2024-04-25T11:37:31.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8004a30 con 0x7f6d080c96e0 2024-04-25T11:37:31.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d080c7500 con 0x7f6d080c96e0 2024-04-25T11:37:31.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0806a350 con 0x7f6d080c96e0 2024-04-25T11:37:31.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6cf801a020 con 0x7f6d080c96e0 2024-04-25T11:37:31.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 0x7f6cec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:37:31.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f6cf808f090 con 0x7f6d080c96e0 2024-04-25T11:37:31.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0d103640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 0x7f6cec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:37:31.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.160+0000 7f6d0d103640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 0x7f6cec071910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009a40 tx=0x7f6d04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:37:31.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.164+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6cf805b990 con 0x7f6d080c96e0 2024-04-25T11:37:31.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.384+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d081579a0 con 0x7f6d080c96e0 2024-04-25T11:37:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.384+0000 7f6cfdffb640 1 -- 172.21.15.136:0/2771928664 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f6cf805b290 con 0x7f6d080c96e0 2024-04-25T11:37:31.369 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:37:31.369 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:37:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 msgr2=0x7f6cec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 0x7f6cec071910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009a40 tx=0x7f6d04008040 comp rx=0 tx=0).stop 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 msgr2=0x7f6d080c5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800c370 tx=0x7f6cf800c830 comp rx=0 tx=0).stop 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 shutdown_connections 2024-04-25T11:37:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d0815aae0 0x7f6d080c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d080c96e0 0x7f6d080c5f30 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6cec06f400 0x7f6cec071910 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 --2- 172.21.15.136:0/2771928664 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d080c79e0 0x7f6d080cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 >> 172.21.15.136:0/2771928664 conn(0x7f6d0806f9d0 msgr2=0x7f6d0806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 shutdown_connections 2024-04-25T11:37:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:37:31.388+0000 7f6d0f367640 1 -- 172.21.15.136:0/2771928664 wait complete. 2024-04-25T11:37:31.393 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-25T11:37:31.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2024-04-25T11:37:54.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:37:54.154 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:38:01.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:01.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 Processor -- start 2024-04-25T11:38:01.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 -- start start 2024-04-25T11:38:01.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd1e0074980 0x7fd1e00c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00c3750 con 0x7fd1e00c4170 2024-04-25T11:38:01.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00c3190 con 0x7fd1e014f840 2024-04-25T11:38:01.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.587+0000 7fd1e4a2d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e006dc00 con 0x7fd1e0074980 2024-04-25T11:38:01.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ded9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd1e00c4170 0x7fd1e01636f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55564/0 (socket says 172.21.15.136:55564) 2024-04-25T11:38:01.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ded9d640 1 -- 172.21.15.136:0/2411161601 learned_addr learned my addr 172.21.15.136:0/2411161601 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/2411161601 >> v2:172.21.15.157:3300/0 conn(0x7fd1e0074980 0x7fd1e00c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344458260 0 0) 0x7fd1e00c3750 con 0x7fd1e00c4170 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c8003530 con 0x7fd1e00c4170 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3185336451 0 0) 0x7fd1c8003530 con 0x7fd1e00c4170 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 >> v2:172.21.15.157:3300/0 conn(0x7fd1e0074980 msgr2=0x7fd1e00c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 --2- 172.21.15.136:0/2411161601 >> v2:172.21.15.157:3300/0 conn(0x7fd1e0074980 0x7fd1e00c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 >> v1:172.21.15.157:6789/0 conn(0x7fd1e014f840 legacy=0x7fd1e0073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00c3470 con 0x7fd1e00c4170 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd1d8001a00 con 0x7fd1e00c4170 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 >> v1:172.21.15.136:6789/0 conn(0x7fd1e00c4170 legacy=0x7fd1e01636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1c80039c0 0x7fd1c8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 0x7fd1c8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c80051d0 0x7fd1c800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e006dc00 con 0x7fd1c8004810 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c800c170 con 0x7fd1c80051d0 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c800e580 con 0x7fd1c80039c0 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 0x7fd1c8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ded9d640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c80051d0 0x7fd1c800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 -- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1c80039c0 msgr2=0x7fd1c8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1de59c640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1c80039c0 0x7fd1c8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1c80039c0 0x7fd1c8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 -- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c80051d0 msgr2=0x7fd1c800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:01.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c80051d0 0x7fd1c800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 -- 172.21.15.136:0/2411161601 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c8003530 con 0x7fd1c8004810 2024-04-25T11:38:01.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ded9d640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c80051d0 0x7fd1c800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:01.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ded9d640 1 -- 172.21.15.136:0/2411161601 reap_dead start 2024-04-25T11:38:01.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1de59c640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1c80039c0 0x7fd1c8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:01.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.591+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 0x7fd1c8004c20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0073070 tx=0x7fd1d401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5f01c0475fd720 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:01.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40b2070 con 0x7fd1c8004810 2024-04-25T11:38:01.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d40a5600 con 0x7fd1c8004810 2024-04-25T11:38:01.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1dd59a640 1 -- 172.21.15.136:0/2411161601 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d40096b0 con 0x7fd1c8004810 2024-04-25T11:38:01.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 msgr2=0x7fd1c8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 0x7fd1c8004c20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0073070 tx=0x7fd1d401ca10 comp rx=0 tx=0).stop 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/2411161601 shutdown_connections 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/2411161601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c8004810 0x7fd1c8004c20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/2411161601 >> 172.21.15.136:0/2411161601 conn(0x7fd1e00ca5a0 msgr2=0x7fd1e00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/2411161601 shutdown_connections 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/2411161601 wait complete. 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 Processor -- start 2024-04-25T11:38:01.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.595+0000 7fd1e4a2d640 1 -- start start 2024-04-25T11:38:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 0x7fd1e014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1e0156380 0x7fd1e0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00c4580 con 0x7fd1e00c59b0 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e014fc50 con 0x7fd1c8004810 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e0159c10 con 0x7fd1e0156380 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35278/0 (socket says 172.21.15.136:35278) 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 -- 172.21.15.136:0/4030153589 learned_addr learned my addr 172.21.15.136:0/4030153589 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ddd9b640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1e0156380 0x7fd1e0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1de59c640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 0x7fd1e014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 -- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1e0156380 msgr2=0x7fd1e0070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1e0156380 0x7fd1e0070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 -- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 msgr2=0x7fd1e014a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 0x7fd1e014a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 -- 172.21.15.136:0/4030153589 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00c7a60 con 0x7fd1e00c59b0 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1de59c640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 0x7fd1e014a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:01.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ded9d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0158db0 tx=0x7fd1d8010cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d801a070 con 0x7fd1e00c59b0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d8011040 con 0x7fd1e00c59b0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00c7d20 con 0x7fd1e00c59b0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800ed10 con 0x7fd1e00c59b0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e00c5220 con 0x7fd1e00c59b0 2024-04-25T11:38:01.585 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e006a350 con 0x7fd1e00c59b0 2024-04-25T11:38:01.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd1d801d070 con 0x7fd1e00c59b0 2024-04-25T11:38:01.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.599+0000 7fd1ceffd640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 0x7fd1c0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:01.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.603+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fd1d80921a0 con 0x7fd1e00c59b0 2024-04-25T11:38:01.586 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.603+0000 7fd1de59c640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 0x7fd1c0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:01.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.603+0000 7fd1de59c640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 0x7fd1c0071940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0000c10 tx=0x7fd1d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:01.590 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.607+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd1d805e980 con 0x7fd1e00c59b0 2024-04-25T11:38:01.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.843+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1e00c2ed0 con 0x7fd1e00c59b0 2024-04-25T11:38:01.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.843+0000 7fd1ceffd640 1 -- 172.21.15.136:0/4030153589 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7fd1d805e280 con 0x7fd1e00c59b0 2024-04-25T11:38:01.828 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:01.828 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:01.828 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:01.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 msgr2=0x7fd1c0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 0x7fd1c0071940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0000c10 tx=0x7fd1d0009040 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 msgr2=0x7fd1e00c5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0158db0 tx=0x7fd1d8010cf0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 shutdown_connections 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e00c59b0 0x7fd1e00c5dc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1c006f430 0x7fd1c0071940 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1e0156380 0x7fd1e0070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 --2- 172.21.15.136:0/4030153589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1c8004810 0x7fd1e014a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 >> 172.21.15.136:0/4030153589 conn(0x7fd1e00ca5a0 msgr2=0x7fd1e014c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 shutdown_connections 2024-04-25T11:38:01.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:01.847+0000 7fd1e4a2d640 1 -- 172.21.15.136:0/4030153589 wait complete. 2024-04-25T11:38:01.850 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2024-04-25T11:38:01.850 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T11:38:01.850 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:38:01.874 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T11:38:01.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:02.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 Processor -- start 2024-04-25T11:38:02.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 -- start start 2024-04-25T11:38:02.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6c8014f840 0x7f6c80074980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c800c3750 con 0x7f6c80073200 2024-04-25T11:38:02.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c800c3190 con 0x7f6c800c4340 2024-04-25T11:38:02.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c887c2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6c8006dc00 con 0x7f6c8014f840 2024-04-25T11:38:02.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c85d5d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6c80073200 0x7f6c800c3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55578/0 (socket says 172.21.15.136:55578) 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c85d5d640 1 -- 172.21.15.136:0/517647805 learned_addr learned my addr 172.21.15.136:0/517647805 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8655e640 1 --2- 172.21.15.136:0/517647805 >> v2:172.21.15.157:3300/0 conn(0x7f6c8014f840 0x7f6c80074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751958316 0 0) 0x7f6c800c3750 con 0x7f6c80073200 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c6c003530 con 0x7f6c80073200 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4128471569 0 0) 0x7f6c6c003530 con 0x7f6c80073200 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 >> v2:172.21.15.157:3300/0 conn(0x7f6c8014f840 msgr2=0x7f6c80074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 --2- 172.21.15.136:0/517647805 >> v2:172.21.15.157:3300/0 conn(0x7f6c8014f840 0x7f6c80074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 >> v1:172.21.15.157:6789/0 conn(0x7f6c800c4340 legacy=0x7f6c801636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c800c3470 con 0x7f6c80073200 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f6c70001680 con 0x7f6c80073200 2024-04-25T11:38:02.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 >> v1:172.21.15.136:6789/0 conn(0x7f6c80073200 legacy=0x7f6c800c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c6c0039c0 0x7f6c6c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c6c004810 0x7f6c6c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c6c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8006dc00 con 0x7f6c6c0051d0 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c00c170 con 0x7f6c6c004810 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c6c00c370 con 0x7f6c6c0039c0 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c6c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c6c004810 0x7f6c6c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.043+0000 7f6c8655e640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c6c0039c0 0x7f6c6c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 -- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c6c0039c0 msgr2=0x7f6c6c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c6c0039c0 0x7f6c6c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 -- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c6c004810 msgr2=0x7f6c6c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:02.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c6c004810 0x7f6c6c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 -- 172.21.15.136:0/517647805 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6c003530 con 0x7f6c6c0051d0 2024-04-25T11:38:02.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c6c004810 0x7f6c6c004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 -- 172.21.15.136:0/517647805 reap_dead start 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c6c00b9d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6c78001e40 tx=0x7f6c7801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748dee99d2218651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78001a90 con 0x7f6c6c0051d0 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c78003b80 con 0x7f6c6c0051d0 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c8555c640 1 -- 172.21.15.136:0/517647805 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c780050d0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 -- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 msgr2=0x7f6c6c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c6c00b9d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6c78001e40 tx=0x7f6c7801c6a0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 -- 172.21.15.136:0/517647805 shutdown_connections 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 --2- 172.21.15.136:0/517647805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c6c00b9d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 -- 172.21.15.136:0/517647805 >> 172.21.15.136:0/517647805 conn(0x7f6c800ca5a0 msgr2=0x7f6c800ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 -- 172.21.15.136:0/517647805 shutdown_connections 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.047+0000 7f6c887c2640 1 -- 172.21.15.136:0/517647805 wait complete. 2024-04-25T11:38:02.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 Processor -- start 2024-04-25T11:38:02.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 -- start start 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 0x7f6c80149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 0x7f6c8006cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6c80153380 con 0x7f6c6c0051d0 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8014fc50 con 0x7f6c801f6400 2024-04-25T11:38:02.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.051+0000 7f6c887c2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c80073610 con 0x7f6c8014a8f0 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35294/0 (socket says 172.21.15.136:35294) 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 -- 172.21.15.136:0/3559899847 learned_addr learned my addr 172.21.15.136:0/3559899847 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 0x7f6c8006cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 0x7f6c80149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 -- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 msgr2=0x7f6c8006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 0x7f6c8006cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 -- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 msgr2=0x7f6c80149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 0x7f6c80149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 -- 172.21.15.136:0/3559899847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8015cf30 con 0x7f6c6c0051d0 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 0x7f6c80149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c86d5f640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 0x7f6c8006cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:02.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c8655e640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c011560 tx=0x7f6c7c011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c015930 con 0x7f6c6c0051d0 2024-04-25T11:38:02.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c7c015ad0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c801f5920 con 0x7f6c6c0051d0 2024-04-25T11:38:02.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c01d400 con 0x7f6c6c0051d0 2024-04-25T11:38:02.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c801f5ee0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6c7c01d5a0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.055+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8006a350 con 0x7f6c6c0051d0 2024-04-25T11:38:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.059+0000 7f6c76ffd640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 0x7f6c64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.059+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f6c7c0901e0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.059+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 0x7f6c64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:02.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.059+0000 7f6c85d5d640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 0x7f6c64071940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c70000fc0 tx=0x7f6c700011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:02.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.063+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6c7c019030 con 0x7f6c6c0051d0 2024-04-25T11:38:02.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.307+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c800c2ed0 con 0x7f6c6c0051d0 2024-04-25T11:38:02.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.307+0000 7f6c76ffd640 1 -- 172.21.15.136:0/3559899847 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f6c7c05ca50 con 0x7f6c6c0051d0 2024-04-25T11:38:02.293 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:02.293 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:02.294 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:02.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 msgr2=0x7f6c64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 0x7f6c64071940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6c70000fc0 tx=0x7f6c700011a0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 msgr2=0x7f6c80149590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c011560 tx=0x7f6c7c011a20 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 shutdown_connections 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6c6406f430 0x7f6c64071940 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8014a8f0 0x7f6c8006cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c801f6400 0x7f6c80149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 --2- 172.21.15.136:0/3559899847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c6c0051d0 0x7f6c80149590 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 >> 172.21.15.136:0/3559899847 conn(0x7f6c800ca5a0 msgr2=0x7f6c8014ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 shutdown_connections 2024-04-25T11:38:02.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:02.311+0000 7f6c887c2640 1 -- 172.21.15.136:0/3559899847 wait complete. 2024-04-25T11:38:02.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:04.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:04.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 Processor -- start 2024-04-25T11:38:04.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 -- start start 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2340c2e00 0x7fb23415bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb2340c4340 0x7fb2340c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2340c3210 con 0x7fb23415c300 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb23415cae0 con 0x7fb2340c4340 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23c564640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2340c9d20 con 0x7fb2340c2e00 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb2340c4340 0x7fb2340c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb2340c4340 0x7fb2340c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35298/0 (socket says 172.21.15.136:35298) 2024-04-25T11:38:04.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 -- 172.21.15.136:0/3329240756 learned_addr learned my addr 172.21.15.136:0/3329240756 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:04.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23a300640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2340c2e00 0x7fb23415bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 -- 172.21.15.136:0/3329240756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2340c2e00 msgr2=0x7fb23415bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:04.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2340c2e00 0x7fb23415bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 -- 172.21.15.136:0/3329240756 >> v1:172.21.15.157:6789/0 conn(0x7fb23415c300 legacy=0x7fb2340749a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.495+0000 7fb23ab01640 1 -- 172.21.15.136:0/3329240756 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340c4c90 con 0x7fb2340c4340 2024-04-25T11:38:04.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23ab01640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c4340 0x7fb2340c4750 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1591450792825f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:04.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb2392fe640 1 -- 172.21.15.136:0/3329240756 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a55d0 con 0x7fb2340c4340 2024-04-25T11:38:04.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb2392fe640 1 -- 172.21.15.136:0/3329240756 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c0a5770 con 0x7fb2340c4340 2024-04-25T11:38:04.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb2392fe640 1 -- 172.21.15.136:0/3329240756 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c0a5b40 con 0x7fb2340c4340 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 -- 172.21.15.136:0/3329240756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c4340 msgr2=0x7fb2340c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c4340 0x7fb2340c4750 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c01c910 comp rx=0 tx=0).stop 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 -- 172.21.15.136:0/3329240756 shutdown_connections 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c4340 0x7fb2340c4750 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 --2- 172.21.15.136:0/3329240756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2340c2e00 0x7fb23415bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 -- 172.21.15.136:0/3329240756 >> 172.21.15.136:0/3329240756 conn(0x7fb2340ca5a0 msgr2=0x7fb2340ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 -- 172.21.15.136:0/3329240756 shutdown_connections 2024-04-25T11:38:04.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.499+0000 7fb23c564640 1 -- 172.21.15.136:0/3329240756 wait complete. 2024-04-25T11:38:04.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 Processor -- start 2024-04-25T11:38:04.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 -- start start 2024-04-25T11:38:04.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2340c4340 0x7fb23415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb23415c300 0x7fb23415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb2340750b0 con 0x7fb2340c2e00 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2340ca000 con 0x7fb2340c4340 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23c564640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb234073200 con 0x7fb23415c300 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35306/0 (socket says 172.21.15.136:35306) 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 -- 172.21.15.136:0/3368508645 learned_addr learned my addr 172.21.15.136:0/3368508645 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23ab01640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb23415c300 0x7fb23415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb239aff640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2340c4340 0x7fb23415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 -- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb23415c300 msgr2=0x7fb23415ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb23415c300 0x7fb23415ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 -- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2340c4340 msgr2=0x7fb23415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2340c4340 0x7fb23415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 -- 172.21.15.136:0/3368508645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb234150480 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb23a300640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb23000e870 tx=0x7fb23000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230017070 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.503+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23000c940 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2340c6b40 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23000ccf0 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23414f8f0 con 0x7fb2340c2e00 2024-04-25T11:38:04.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23406a350 con 0x7fb2340c2e00 2024-04-25T11:38:04.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb23001a070 con 0x7fb2340c2e00 2024-04-25T11:38:04.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb22affd640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:04.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fb230090000 con 0x7fb2340c2e00 2024-04-25T11:38:04.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.507+0000 7fb239aff640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:04.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.511+0000 7fb239aff640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 0x7fb218071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb2340716f0 tx=0x7fb224009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:04.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.511+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb23005c8c0 con 0x7fb2340c2e00 2024-04-25T11:38:04.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.711+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2341560c0 con 0x7fb2340c2e00 2024-04-25T11:38:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.711+0000 7fb22affd640 1 -- 172.21.15.136:0/3368508645 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7fb23005c1c0 con 0x7fb2340c2e00 2024-04-25T11:38:04.697 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:04.697 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:04.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 msgr2=0x7fb218071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 0x7fb218071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb2340716f0 tx=0x7fb224009040 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 msgr2=0x7fb2340c6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb23000e870 tx=0x7fb23000ec70 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 shutdown_connections 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb23415c300 0x7fb23415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2340c4340 0x7fb23415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 --2- 172.21.15.136:0/3368508645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2340c2e00 0x7fb2340c6e90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 >> 172.21.15.136:0/3368508645 conn(0x7fb2340ca5a0 msgr2=0x7fb2340cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 shutdown_connections 2024-04-25T11:38:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:04.715+0000 7fb23c564640 1 -- 172.21.15.136:0/3368508645 wait complete. 2024-04-25T11:38:04.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:06.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:06.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 Processor -- start 2024-04-25T11:38:06.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 -- start start 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 0x7f96e0152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96e0158590 0x7f96e01589a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e0071300 con 0x7f96e00736a0 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f96e00710d0 con 0x7f96e0158590 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e85b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00c95f0 con 0x7f96e00c4750 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96e0158590 0x7f96e01589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f96e0158590 0x7f96e01589a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43352/0 (socket says 172.21.15.136:43352) 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 -- 172.21.15.136:0/1658382833 learned_addr learned my addr 172.21.15.136:0/1658382833 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e6352640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 0x7f96e0152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 -- 172.21.15.136:0/1658382833 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 msgr2=0x7f96e0152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 0x7f96e0152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:06.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 -- 172.21.15.136:0/1658382833 >> v1:172.21.15.157:6789/0 conn(0x7f96e00736a0 legacy=0x7f96e015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:06.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.895+0000 7f96e5b51640 1 -- 172.21.15.136:0/1658382833 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e00c7300 con 0x7f96e0158590 2024-04-25T11:38:06.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e6352640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 0x7f96e0152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:06.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e5b51640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e0158590 0x7f96e01589a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009870 tx=0x7f96d001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf85308304562228 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:06.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e5350640 1 -- 172.21.15.136:0/1658382833 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d00b0070 con 0x7f96e0158590 2024-04-25T11:38:06.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e5350640 1 -- 172.21.15.136:0/1658382833 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d00ab600 con 0x7f96e0158590 2024-04-25T11:38:06.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e5350640 1 -- 172.21.15.136:0/1658382833 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d00ab9d0 con 0x7f96e0158590 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e85b6640 1 -- 172.21.15.136:0/1658382833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e0158590 msgr2=0x7f96e01589a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.899+0000 7f96e85b6640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e0158590 0x7f96e01589a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009870 tx=0x7f96d001c440 comp rx=0 tx=0).stop 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- 172.21.15.136:0/1658382833 shutdown_connections 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e0158590 0x7f96e01589a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 --2- 172.21.15.136:0/1658382833 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00c4750 0x7f96e0152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- 172.21.15.136:0/1658382833 >> 172.21.15.136:0/1658382833 conn(0x7f96e00c9d50 msgr2=0x7f96e00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- 172.21.15.136:0/1658382833 shutdown_connections 2024-04-25T11:38:06.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- 172.21.15.136:0/1658382833 wait complete. 2024-04-25T11:38:06.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 Processor -- start 2024-04-25T11:38:06.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- start start 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00736a0 0x7f96e01f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96e0158590 0x7f96e0147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00ca870 con 0x7f96e00c4750 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00c9a80 con 0x7f96e0158590 2024-04-25T11:38:06.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e85b6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00ca9f0 con 0x7f96e00736a0 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43360/0 (socket says 172.21.15.136:43360) 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 -- 172.21.15.136:0/3403463116 learned_addr learned my addr 172.21.15.136:0/3403463116 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e6352640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00736a0 0x7f96e01f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e6b53640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96e0158590 0x7f96e0147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 -- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00736a0 msgr2=0x7f96e01f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00736a0 0x7f96e01f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 -- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96e0158590 msgr2=0x7f96e0147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96e0158590 0x7f96e0147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.903+0000 7f96e5b51640 1 -- 172.21.15.136:0/3403463116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e01f40f0 con 0x7f96e00c4750 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e5b51640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f96d00ae040 tx=0x7f96d001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d00ba020 con 0x7f96e00c4750 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d00b6620 con 0x7f96e00c4750 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e014fe60 con 0x7f96e00c4750 2024-04-25T11:38:06.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d00b0040 con 0x7f96e00c4750 2024-04-25T11:38:06.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e014fb80 con 0x7f96e00c4750 2024-04-25T11:38:06.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e006a350 con 0x7f96e00c4750 2024-04-25T11:38:06.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f96d00bd020 con 0x7f96e00c4750 2024-04-25T11:38:06.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 0x7f96c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:06.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f96d0129c30 con 0x7f96e00c4750 2024-04-25T11:38:06.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e6352640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 0x7f96c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:06.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.907+0000 7f96e6352640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 0x7f96c4071910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ba10 tx=0x7f96dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:06.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:06.911+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f96d00c4030 con 0x7f96e00c4750 2024-04-25T11:38:07.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.103+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96e0070eb0 con 0x7f96e00c4750 2024-04-25T11:38:07.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.103+0000 7f96d6ffd640 1 -- 172.21.15.136:0/3403463116 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f96d0016150 con 0x7f96e00c4750 2024-04-25T11:38:07.090 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:07.090 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:07.090 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:07.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 msgr2=0x7f96c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 0x7f96c4071910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00ba10 tx=0x7f96dc008040 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 msgr2=0x7f96e0147aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f96d00ae040 tx=0x7f96d001ac80 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 shutdown_connections 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96e0158590 0x7f96e0147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96e00c4750 0x7f96e0147aa0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f96c406f400 0x7f96c4071910 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 --2- 172.21.15.136:0/3403463116 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96e00736a0 0x7f96e01f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 >> 172.21.15.136:0/3403463116 conn(0x7f96e00c9d50 msgr2=0x7f96e0156da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 shutdown_connections 2024-04-25T11:38:07.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:07.107+0000 7f96e85b6640 1 -- 172.21.15.136:0/3403463116 wait complete. 2024-04-25T11:38:07.109 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:09.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:09.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.287+0000 7f2c350db640 1 Processor -- start 2024-04-25T11:38:09.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.287+0000 7f2c350db640 1 -- start start 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c350db640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2c30154060 0x7f2c300c8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c350db640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c3015a5f0 0x7f2c300c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c350db640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c300c8d40 con 0x7f2c3015aa00 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c350db640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2c300c8fd0 con 0x7f2c3015a5f0 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c350db640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2c300c3880 con 0x7f2c30154060 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c3015a5f0 0x7f2c300c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2f59e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2c3015aa00 0x7f2c3015d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:57954/0 (socket says 172.21.15.136:57954) 2024-04-25T11:38:09.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2f59e640 1 -- 172.21.15.136:0/1797226222 learned_addr learned my addr 172.21.15.136:0/1797226222 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2ed9d640 1 --2- 172.21.15.136:0/1797226222 >> v2:172.21.15.157:3300/0 conn(0x7f2c30154060 0x7f2c300c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 -- 172.21.15.136:0/1797226222 >> v2:172.21.15.157:3300/0 conn(0x7f2c30154060 msgr2=0x7f2c300c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 --2- 172.21.15.136:0/1797226222 >> v2:172.21.15.157:3300/0 conn(0x7f2c30154060 0x7f2c300c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 -- 172.21.15.136:0/1797226222 >> v1:172.21.15.136:6789/0 conn(0x7f2c3015aa00 legacy=0x7f2c3015d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 -- 172.21.15.136:0/1797226222 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3006d9b0 con 0x7f2c3015a5f0 2024-04-25T11:38:09.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2e59c640 1 -- 172.21.15.136:0/1797226222 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430211379 0 0) 0x7f2c300c8d40 con 0x7f2c3015aa00 2024-04-25T11:38:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2659c640 1 --2- 172.21.15.136:0/1797226222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c3015a5f0 0x7f2c300c71b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2c300c7c60 tx=0x7f2c1801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc14f4871732d3c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2e59c640 1 -- 172.21.15.136:0/1797226222 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180b5070 con 0x7f2c3015a5f0 2024-04-25T11:38:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2e59c640 1 -- 172.21.15.136:0/1797226222 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c180a8970 con 0x7f2c3015a5f0 2024-04-25T11:38:09.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.291+0000 7f2c2e59c640 1 -- 172.21.15.136:0/1797226222 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180a8d40 con 0x7f2c3015a5f0 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- 172.21.15.136:0/1797226222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c3015a5f0 msgr2=0x7f2c300c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 --2- 172.21.15.136:0/1797226222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c3015a5f0 0x7f2c300c71b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2c300c7c60 tx=0x7f2c1801ec70 comp rx=0 tx=0).stop 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- 172.21.15.136:0/1797226222 shutdown_connections 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 --2- 172.21.15.136:0/1797226222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c3015a5f0 0x7f2c300c71b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 --2- 172.21.15.136:0/1797226222 >> v2:172.21.15.157:3300/0 conn(0x7f2c30154060 0x7f2c300c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- 172.21.15.136:0/1797226222 >> 172.21.15.136:0/1797226222 conn(0x7f2c300ca5a0 msgr2=0x7f2c300ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- 172.21.15.136:0/1797226222 shutdown_connections 2024-04-25T11:38:09.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- 172.21.15.136:0/1797226222 wait complete. 2024-04-25T11:38:09.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 Processor -- start 2024-04-25T11:38:09.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.295+0000 7f2c350db640 1 -- start start 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c3015a5f0 0x7f2c301f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c3015aa00 0x7f2c30075ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3006df30 con 0x7f2c30154060 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30154470 con 0x7f2c3015a5f0 2024-04-25T11:38:09.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c3015a0d0 con 0x7f2c3015aa00 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43378/0 (socket says 172.21.15.136:43378) 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 -- 172.21.15.136:0/2809212240 learned_addr learned my addr 172.21.15.136:0/2809212240 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2659c640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c3015a5f0 0x7f2c301f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2f59e640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c3015aa00 0x7f2c30075ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 -- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c3015aa00 msgr2=0x7f2c30075ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c3015aa00 0x7f2c30075ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 -- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c3015a5f0 msgr2=0x7f2c301f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c3015a5f0 0x7f2c301f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 -- 172.21.15.136:0/2809212240 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c301f0ed0 con 0x7f2c30154060 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2ed9d640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000d620 tx=0x7f2c2000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000b620 con 0x7f2c30154060 2024-04-25T11:38:09.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c20019070 con 0x7f2c30154060 2024-04-25T11:38:09.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000b9e0 con 0x7f2c30154060 2024-04-25T11:38:09.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3015b470 con 0x7f2c30154060 2024-04-25T11:38:09.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c3015bf80 con 0x7f2c30154060 2024-04-25T11:38:09.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3006a350 con 0x7f2c30154060 2024-04-25T11:38:09.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2c2000bb80 con 0x7f2c30154060 2024-04-25T11:38:09.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 0x7f2c10071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:09.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f2c2008fd30 con 0x7f2c30154060 2024-04-25T11:38:09.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.299+0000 7f2c2659c640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 0x7f2c10071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:09.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.303+0000 7f2c2659c640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 0x7f2c10071910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2c180b2000 tx=0x7f2c180aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:09.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.307+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2c301f0ed0 con 0x7f2c30154060 2024-04-25T11:38:09.486 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.499+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c300c85e0 con 0x7f2c30154060 2024-04-25T11:38:09.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c27fff640 1 -- 172.21.15.136:0/2809212240 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f2c2005c630 con 0x7f2c30154060 2024-04-25T11:38:09.487 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:09.487 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:09.487 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:09.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 msgr2=0x7f2c10071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 0x7f2c10071910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2c180b2000 tx=0x7f2c180aa070 comp rx=0 tx=0).stop 2024-04-25T11:38:09.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 msgr2=0x7f2c3015ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000d620 tx=0x7f2c2000dae0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 shutdown_connections 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2c1006f400 0x7f2c10071910 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c3015aa00 0x7f2c30075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c3015a5f0 0x7f2c301f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 --2- 172.21.15.136:0/2809212240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c30154060 0x7f2c3015ca60 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 >> 172.21.15.136:0/2809212240 conn(0x7f2c300ca5a0 msgr2=0x7f2c3014d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 shutdown_connections 2024-04-25T11:38:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:09.503+0000 7f2c350db640 1 -- 172.21.15.136:0/2809212240 wait complete. 2024-04-25T11:38:09.509 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:11.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:11.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 Processor -- start 2024-04-25T11:38:11.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 -- start start 2024-04-25T11:38:11.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc8fc06d780 0x7fc8fc160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8fc0ca9b0 con 0x7fc8fc0ca500 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc8fc06eff0 con 0x7fc8fc0cf490 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc901993640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc8fc161470 con 0x7fc8fc06d780 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43392/0 (socket says 172.21.15.136:43392) 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1701251459 learned_addr learned my addr 172.21.15.136:0/1701251459 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fa7fc640 1 --2- 172.21.15.136:0/1701251459 >> v2:172.21.15.157:3300/0 conn(0x7fc8fc06d780 0x7fc8fc160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1701251459 >> v2:172.21.15.157:3300/0 conn(0x7fc8fc06d780 msgr2=0x7fc8fc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 --2- 172.21.15.136:0/1701251459 >> v2:172.21.15.157:3300/0 conn(0x7fc8fc06d780 0x7fc8fc160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1701251459 >> v1:172.21.15.157:6789/0 conn(0x7fc8fc0ca500 legacy=0x7fc8fc0c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.687+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1701251459 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc06e230 con 0x7fc8fc0cf490 2024-04-25T11:38:11.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc8fb7fe640 1 --2- 172.21.15.136:0/1701251459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc074b30 tx=0x7fc8ec01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58da6000ee41a507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:11.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc8f9ffb640 1 -- 172.21.15.136:0/1701251459 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0b0070 con 0x7fc8fc0cf490 2024-04-25T11:38:11.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc8f9ffb640 1 -- 172.21.15.136:0/1701251459 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0ab600 con 0x7fc8fc0cf490 2024-04-25T11:38:11.676 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc8f9ffb640 1 -- 172.21.15.136:0/1701251459 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0ab9d0 con 0x7fc8fc0cf490 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 -- 172.21.15.136:0/1701251459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 msgr2=0x7fc8fc0cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 --2- 172.21.15.136:0/1701251459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc074b30 tx=0x7fc8ec01c3a0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 -- 172.21.15.136:0/1701251459 shutdown_connections 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 --2- 172.21.15.136:0/1701251459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc0cf8a0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 --2- 172.21.15.136:0/1701251459 >> v2:172.21.15.157:3300/0 conn(0x7fc8fc06d780 0x7fc8fc160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 -- 172.21.15.136:0/1701251459 >> 172.21.15.136:0/1701251459 conn(0x7fc8fc070f70 msgr2=0x7fc8fc068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 -- 172.21.15.136:0/1701251459 shutdown_connections 2024-04-25T11:38:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.691+0000 7fc901993640 1 -- 172.21.15.136:0/1701251459 wait complete. 2024-04-25T11:38:11.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 Processor -- start 2024-04-25T11:38:11.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- start start 2024-04-25T11:38:11.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8fc06d780 0x7fc8fc1fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 0x7fc8fc1fa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc0c5a10 con 0x7fc8fc0cf490 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc0c5890 con 0x7fc8fc0ca500 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8fc16d570 con 0x7fc8fc06d780 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43394/0 (socket says 172.21.15.136:43394) 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1171773153 learned_addr learned my addr 172.21.15.136:0/1171773153 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fa7fc640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 0x7fc8fc1fa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8faffd640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8fc06d780 0x7fc8fc1fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8fc06d780 msgr2=0x7fc8fc1fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8fc06d780 0x7fc8fc1fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 msgr2=0x7fc8fc1fa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 0x7fc8fc1fa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 -- 172.21.15.136:0/1171773153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc14ad80 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fa7fc640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 0x7fc8fc1fa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8fb7fe640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0ae040 tx=0x7fc8ec01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0ba020 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ec0b6420 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8fc1fab30 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec0b0040 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.695+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8fc1fddc0 con 0x7fc8fc0cf490 2024-04-25T11:38:11.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8fc14af50 con 0x7fc8fc0cf490 2024-04-25T11:38:11.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc8ec0bd020 con 0x7fc8fc0cf490 2024-04-25T11:38:11.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc8f3fff640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 0x7fc8dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:11.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7fc8ec12a070 con 0x7fc8fc0cf490 2024-04-25T11:38:11.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc8faffd640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 0x7fc8dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:11.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.699+0000 7fc8faffd640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 0x7fc8dc071910 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4007bf0 tx=0x7fc8f4007240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:11.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.703+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc8ec0c4070 con 0x7fc8fc0cf490 2024-04-25T11:38:11.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.895+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8fc1fd800 con 0x7fc8fc0cf490 2024-04-25T11:38:11.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.895+0000 7fc8f3fff640 1 -- 172.21.15.136:0/1171773153 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7fc8ec0160b0 con 0x7fc8fc0cf490 2024-04-25T11:38:11.880 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:11.880 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:11.880 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.895+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 msgr2=0x7fc8dc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.895+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 0x7fc8dc071910 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4007bf0 tx=0x7fc8f4007240 comp rx=0 tx=0).stop 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 msgr2=0x7fc8fc14a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0ae040 tx=0x7fc8ec01abe0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 shutdown_connections 2024-04-25T11:38:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8fc0cf490 0x7fc8fc14a1f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8fc0ca500 0x7fc8fc1fa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc8dc06f400 0x7fc8dc071910 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 --2- 172.21.15.136:0/1171773153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8fc06d780 0x7fc8fc1fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 >> 172.21.15.136:0/1171773153 conn(0x7fc8fc070f70 msgr2=0x7fc8fc160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 shutdown_connections 2024-04-25T11:38:11.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:11.899+0000 7fc901993640 1 -- 172.21.15.136:0/1171773153 wait complete. 2024-04-25T11:38:11.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:13.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:14.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 Processor -- start 2024-04-25T11:38:14.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 -- start start 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f147c070870 0x7f147c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f147c070c80 0x7f147c0c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147c143580 con 0x7f147c143170 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f147c15dd20 con 0x7f147c070c80 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f1482142640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f147c15e000 con 0x7f147c070870 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f147c070c80 0x7f147c0c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f147c070c80 0x7f147c0c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43402/0 (socket says 172.21.15.136:43402) 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 -- 172.21.15.136:0/1053623357 learned_addr learned my addr 172.21.15.136:0/1053623357 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147affd640 1 --2- 172.21.15.136:0/1053623357 >> v2:172.21.15.157:3300/0 conn(0x7f147c070870 0x7f147c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 -- 172.21.15.136:0/1053623357 >> v2:172.21.15.157:3300/0 conn(0x7f147c070870 msgr2=0x7f147c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:14.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 --2- 172.21.15.136:0/1053623357 >> v2:172.21.15.157:3300/0 conn(0x7f147c070870 0x7f147c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 -- 172.21.15.136:0/1053623357 >> v1:172.21.15.157:6789/0 conn(0x7f147c143170 legacy=0x7f147c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.079+0000 7f147bfff640 1 -- 172.21.15.136:0/1053623357 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c06fd90 con 0x7f147c070c80 2024-04-25T11:38:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f147bfff640 1 --2- 172.21.15.136:0/1053623357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c0c6ea0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f147c157a80 tx=0x7f147401efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593323987400b5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f147a7fc640 1 -- 172.21.15.136:0/1053623357 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740a8bd0 con 0x7f147c070c80 2024-04-25T11:38:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f147a7fc640 1 -- 172.21.15.136:0/1053623357 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14740a8d70 con 0x7f147c070c80 2024-04-25T11:38:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f147a7fc640 1 -- 172.21.15.136:0/1053623357 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740b0610 con 0x7f147c070c80 2024-04-25T11:38:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 -- 172.21.15.136:0/1053623357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 msgr2=0x7f147c0c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:14.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 --2- 172.21.15.136:0/1053623357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c0c6ea0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f147c157a80 tx=0x7f147401efc0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 -- 172.21.15.136:0/1053623357 shutdown_connections 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 --2- 172.21.15.136:0/1053623357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c0c6ea0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 --2- 172.21.15.136:0/1053623357 >> v2:172.21.15.157:3300/0 conn(0x7f147c070870 0x7f147c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 -- 172.21.15.136:0/1053623357 >> 172.21.15.136:0/1053623357 conn(0x7f147c0c9d50 msgr2=0x7f147c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 -- 172.21.15.136:0/1053623357 shutdown_connections 2024-04-25T11:38:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.083+0000 7f1482142640 1 -- 172.21.15.136:0/1053623357 wait complete. 2024-04-25T11:38:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 Processor -- start 2024-04-25T11:38:14.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- start start 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 0x7f147c15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 0x7f147c1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f147c15e2e0 con 0x7f147c070c80 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f147c070350 con 0x7f147c070870 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f147c0712a0 con 0x7f147c143170 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43406/0 (socket says 172.21.15.136:43406) 2024-04-25T11:38:14.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 -- 172.21.15.136:0/2397217220 learned_addr learned my addr 172.21.15.136:0/2397217220 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147b7fe640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 0x7f147c15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147bfff640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 0x7f147c1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 -- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 msgr2=0x7f147c1f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 0x7f147c1f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 -- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 msgr2=0x7f147c15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 0x7f147c15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 -- 172.21.15.136:0/2397217220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c1f0440 con 0x7f147c070c80 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147b7fe640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 0x7f147c15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147bfff640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 0x7f147c1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f147affd640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f146c00d800 tx=0x7f146c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c00b6c0 con 0x7f147c070c80 2024-04-25T11:38:14.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146c019070 con 0x7f147c070c80 2024-04-25T11:38:14.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147c1f06c0 con 0x7f147c070c80 2024-04-25T11:38:14.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c00ba80 con 0x7f147c070c80 2024-04-25T11:38:14.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147c1f0c80 con 0x7f147c070c80 2024-04-25T11:38:14.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.087+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f147c06a350 con 0x7f147c070c80 2024-04-25T11:38:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.091+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f146c00bc20 con 0x7f147c070c80 2024-04-25T11:38:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.091+0000 7f1463fff640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 0x7f1468071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.091+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f146c08fdb0 con 0x7f147c070c80 2024-04-25T11:38:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.091+0000 7f147b7fe640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 0x7f1468071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.091+0000 7f147b7fe640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 0x7f1468071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f147000a850 tx=0x7f1470008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.095+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f147c1f0440 con 0x7f147c070c80 2024-04-25T11:38:14.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.283+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f147c0c4000 con 0x7f147c070c80 2024-04-25T11:38:14.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1463fff640 1 -- 172.21.15.136:0/2397217220 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f146c05c620 con 0x7f147c070c80 2024-04-25T11:38:14.270 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:14.270 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:14.270 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 msgr2=0x7f1468071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 0x7f1468071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f147000a850 tx=0x7f1470008040 comp rx=0 tx=0).stop 2024-04-25T11:38:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 msgr2=0x7f147c137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f146c00d800 tx=0x7f146c00dcc0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 shutdown_connections 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f146806f400 0x7f1468071910 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f147c143170 0x7f147c1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f147c070c80 0x7f147c137e30 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 --2- 172.21.15.136:0/2397217220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f147c070870 0x7f147c15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 >> 172.21.15.136:0/2397217220 conn(0x7f147c0c9d50 msgr2=0x7f147c142390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 shutdown_connections 2024-04-25T11:38:14.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:14.287+0000 7f1482142640 1 -- 172.21.15.136:0/2397217220 wait complete. 2024-04-25T11:38:14.289 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:16.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:16.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 Processor -- start 2024-04-25T11:38:16.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 -- start start 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67f40cdee0 0x7f67f40ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f67f40b5490 0x7f67f40b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f40ccb80 con 0x7f67f4156330 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f67f41602f0 con 0x7f67f40cdee0 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f8f4e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f67f4156740 con 0x7f67f40b5490 2024-04-25T11:38:16.449 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67f40cdee0 0x7f67f40ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67f40cdee0 0x7f67f40ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45372/0 (socket says 172.21.15.136:45372) 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3839080465 learned_addr learned my addr 172.21.15.136:0/3839080465 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f2d9d640 1 --2- 172.21.15.136:0/3839080465 >> v2:172.21.15.157:3300/0 conn(0x7f67f40b5490 0x7f67f40b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3839080465 >> v2:172.21.15.157:3300/0 conn(0x7f67f40b5490 msgr2=0x7f67f40b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 --2- 172.21.15.136:0/3839080465 >> v2:172.21.15.157:3300/0 conn(0x7f67f40b5490 0x7f67f40b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3839080465 >> v1:172.21.15.157:6789/0 conn(0x7f67f4156330 legacy=0x7f67f415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:16.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.462+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3839080465 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f415f6e0 con 0x7f67f40cdee0 2024-04-25T11:38:16.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f1d9b640 1 --2- 172.21.15.136:0/3839080465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40ce2f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f67f40b7120 tx=0x7f67e801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ac5737727556c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f159a640 1 -- 172.21.15.136:0/3839080465 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80b5070 con 0x7f67f40cdee0 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f159a640 1 -- 172.21.15.136:0/3839080465 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e80a89d0 con 0x7f67f40cdee0 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f159a640 1 -- 172.21.15.136:0/3839080465 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80a8da0 con 0x7f67f40cdee0 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3839080465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 msgr2=0x7f67f40ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3839080465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40ce2f0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f67f40b7120 tx=0x7f67e801ecd0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3839080465 shutdown_connections 2024-04-25T11:38:16.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3839080465 >> v2:172.21.15.157:3300/0 conn(0x7f67f40b5490 0x7f67f40b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3839080465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40ce2f0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3839080465 >> 172.21.15.136:0/3839080465 conn(0x7f67f406d6c0 msgr2=0x7f67f406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:16.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3839080465 shutdown_connections 2024-04-25T11:38:16.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.466+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3839080465 wait complete. 2024-04-25T11:38:16.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 Processor -- start 2024-04-25T11:38:16.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- start start 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 0x7f67f41572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67f4156330 0x7f67f41f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f67f415ff80 con 0x7f67f40cdee0 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67f415fca0 con 0x7f67f40b5490 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67f40b5de0 con 0x7f67f4156330 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45376/0 (socket says 172.21.15.136:45376) 2024-04-25T11:38:16.457 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3530764640 learned_addr learned my addr 172.21.15.136:0/3530764640 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f259c640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 0x7f67f41572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67f4156330 msgr2=0x7f67f41f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67f4156330 0x7f67f41f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 msgr2=0x7f67f41572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 0x7f67f41572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 -- 172.21.15.136:0/3530764640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f4157810 con 0x7f67f40cdee0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f259c640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 0x7f67f41572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f1d9b640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f67e80b3040 tx=0x7f67e801ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80c0070 con 0x7f67f40cdee0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e80b0a50 con 0x7f67f40cdee0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80b5070 con 0x7f67f40cdee0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f41588f0 con 0x7f67f40cdee0 2024-04-25T11:38:16.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.470+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f40d1e60 con 0x7f67f40cdee0 2024-04-25T11:38:16.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f406a350 con 0x7f67f40cdee0 2024-04-25T11:38:16.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f67e80ba3e0 con 0x7f67f40cdee0 2024-04-25T11:38:16.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67e2ffd640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 0x7f67d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:16.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f67e812cc00 con 0x7f67f40cdee0 2024-04-25T11:38:16.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67f259c640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 0x7f67d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:16.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.474+0000 7f67f259c640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 0x7f67d4071910 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00a850 tx=0x7f67ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:16.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.478+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f67e8018830 con 0x7f67f40cdee0 2024-04-25T11:38:16.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.674+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67f4075cf0 con 0x7f67f40cdee0 2024-04-25T11:38:16.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.674+0000 7f67e2ffd640 1 -- 172.21.15.136:0/3530764640 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+6931 (secure 0 0 0) 0x7f67e80f9460 con 0x7f67f40cdee0 2024-04-25T11:38:16.661 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:16.661 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5408,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:37:20.294547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5356,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5356":{"gid":5356,"name":"d","rank":0,"incarnation":50,"state":"up:active","state_seq":29,"addr":"172.21.15.157:6836/3463942074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3463942074},{"type":"v1","addr":"172.21.15.157:6836","nonce":3463942074}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5356,"qdb_cluster":[6246,5517,5449,5385,5356]},"id":1}]} 2024-04-25T11:38:16.661 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 73 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 msgr2=0x7f67d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 0x7f67d4071910 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00a850 tx=0x7f67ec008040 comp rx=0 tx=0).stop 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 msgr2=0x7f67f40cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f67e80b3040 tx=0x7f67e801ba70 comp rx=0 tx=0).stop 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 shutdown_connections 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f67d406f400 0x7f67d4071910 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67f4156330 0x7f67f41f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f40cdee0 0x7f67f40cf250 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 --2- 172.21.15.136:0/3530764640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67f40b5490 0x7f67f41572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:16.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 >> 172.21.15.136:0/3530764640 conn(0x7f67f406d6c0 msgr2=0x7f67f4152f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:16.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 shutdown_connections 2024-04-25T11:38:16.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:16.678+0000 7f67f8f4e640 1 -- 172.21.15.136:0/3530764640 wait complete. 2024-04-25T11:38:16.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T11:38:18.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:18.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.854+0000 7f672eab6640 1 Processor -- start 2024-04-25T11:38:18.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.854+0000 7f672eab6640 1 -- start start 2024-04-25T11:38:18.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672eab6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6728153190 0x7f67280c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672eab6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67280c8090 con 0x7f67281594b0 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672eab6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67280c6aa0 con 0x7f67280c9570 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672eab6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f67280c8370 con 0x7f6728153190 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672c852640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6728153190 0x7f67280c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672d053640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f67281594b0 0x7f6728163660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36324/0 (socket says 172.21.15.136:36324) 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672d053640 1 -- 172.21.15.136:0/30996643 learned_addr learned my addr 172.21.15.136:0/30996643 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f67277fe640 1 -- 172.21.15.136:0/30996643 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567143953 0 0) 0x7f67280c8090 con 0x7f67281594b0 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f67277fe640 1 -- 172.21.15.136:0/30996643 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6710003530 con 0x7f67281594b0 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672c852640 1 -- 172.21.15.136:0/30996643 >> v1:172.21.15.157:6789/0 conn(0x7f67280c9570 legacy=0x7f6728158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672c852640 1 -- 172.21.15.136:0/30996643 >> v1:172.21.15.136:6789/0 conn(0x7f67281594b0 legacy=0x7f6728163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.858+0000 7f672c852640 1 -- 172.21.15.136:0/30996643 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67280c6d80 con 0x7f6728153190 2024-04-25T11:38:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672c852640 1 --2- 172.21.15.136:0/30996643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6728153190 0x7f67280c9030 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6718009d90 tx=0x7f671801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321966f77b0920bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f67277fe640 1 -- 172.21.15.136:0/30996643 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180b4070 con 0x7f6728153190 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f67277fe640 1 -- 172.21.15.136:0/30996643 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67180af600 con 0x7f6728153190 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f67277fe640 1 -- 172.21.15.136:0/30996643 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180af9d0 con 0x7f6728153190 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- 172.21.15.136:0/30996643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6728153190 msgr2=0x7f67280c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 --2- 172.21.15.136:0/30996643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6728153190 0x7f67280c9030 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6718009d90 tx=0x7f671801f150 comp rx=0 tx=0).stop 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- 172.21.15.136:0/30996643 shutdown_connections 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 --2- 172.21.15.136:0/30996643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6728153190 0x7f67280c9030 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- 172.21.15.136:0/30996643 >> 172.21.15.136:0/30996643 conn(0x7f67280c9d50 msgr2=0x7f67280ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- 172.21.15.136:0/30996643 shutdown_connections 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- 172.21.15.136:0/30996643 wait complete. 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 Processor -- start 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.862+0000 7f672eab6640 1 -- start start 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 0x7f672814a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 0x7f672814f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f67280ca870 con 0x7f67280c9570 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67280c7340 con 0x7f67281594b0 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67280ca9f0 con 0x7f6728153190 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45394/0 (socket says 172.21.15.136:45394) 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 -- 172.21.15.136:0/1809265529 learned_addr learned my addr 172.21.15.136:0/1809265529 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672d053640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 0x7f672814f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 -- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 msgr2=0x7f672814a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f6727fff640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 0x7f672814a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 0x7f672814a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 -- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 msgr2=0x7f672814f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 0x7f672814f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 -- 172.21.15.136:0/1809265529 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672815a720 con 0x7f67280c9570 2024-04-25T11:38:18.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672d053640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 0x7f672814f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f6727fff640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 0x7f672814a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672c852640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f671801bd60 tx=0x7f671801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180b4040 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67180c1070 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6728076080 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718009450 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281f4090 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672806a350 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f67180ba690 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 0x7f6714071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7509+0+0 (secure 0 0 0) 0x7f671812e1d0 con 0x7f67280c9570 2024-04-25T11:38:18.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.866+0000 7f6727fff640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 0x7f6714071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:18.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.870+0000 7f6727fff640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 0x7f6714071910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6728156760 tx=0x7f671c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:18.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:18.874+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6718018cb0 con 0x7f67280c9570 2024-04-25T11:38:19.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.074+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67280c8910 con 0x7f67280c9570 2024-04-25T11:38:19.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.074+0000 7f67257fa640 1 -- 172.21.15.136:0/1809265529 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6098 (secure 0 0 0) 0x7f67180faad0 con 0x7f67280c9570 2024-04-25T11:38:19.062 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:19.062 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:18.992694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:19.062 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 74 2024-04-25T11:38:19.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 msgr2=0x7f6714071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 0x7f6714071910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6728156760 tx=0x7f671c009040 comp rx=0 tx=0).stop 2024-04-25T11:38:19.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 msgr2=0x7f672815a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f671801bd60 tx=0x7f671801bef0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 shutdown_connections 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281594b0 0x7f672814f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f671406f400 0x7f6714071910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6728153190 0x7f672814a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 --2- 172.21.15.136:0/1809265529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67280c9570 0x7f672815a1e0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 >> 172.21.15.136:0/1809265529 conn(0x7f67280c9d50 msgr2=0x7f67280729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 shutdown_connections 2024-04-25T11:38:19.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.078+0000 7f672eab6640 1 -- 172.21.15.136:0/1809265529 wait complete. 2024-04-25T11:38:19.090 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-04-25T11:38:19.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:38:19.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:19.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.250+0000 7f9cceb69640 1 Processor -- start 2024-04-25T11:38:19.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.250+0000 7f9cceb69640 1 -- start start 2024-04-25T11:38:19.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9cc815ab40 0x7f9cc80ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc815d750 con 0x7f9cc80ca290 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc815da30 con 0x7f9cc815ab40 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc815dd10 con 0x7f9cc80b5060 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9cc815ab40 0x7f9cc80ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cc7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9cc80ca290 0x7f9cc815a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36336/0 (socket says 172.21.15.136:36336) 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cc7fff640 1 -- 172.21.15.136:0/1299099025 learned_addr learned my addr 172.21.15.136:0/1299099025 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccc905640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 -- 172.21.15.136:0/1299099025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 msgr2=0x7f9cc80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 -- 172.21.15.136:0/1299099025 >> v1:172.21.15.136:6789/0 conn(0x7f9cc80ca290 legacy=0x7f9cc815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 -- 172.21.15.136:0/1299099025 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc815dff0 con 0x7f9cc815ab40 2024-04-25T11:38:19.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9ccd106640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc815ab40 0x7f9cc80ca6a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009d90 tx=0x7f9cc001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af44c7eedc3c912 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:19.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cc77fe640 1 -- 172.21.15.136:0/1299099025 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00b4070 con 0x7f9cc815ab40 2024-04-25T11:38:19.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cc77fe640 1 -- 172.21.15.136:0/1299099025 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc00af600 con 0x7f9cc815ab40 2024-04-25T11:38:19.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cc77fe640 1 -- 172.21.15.136:0/1299099025 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00af9d0 con 0x7f9cc815ab40 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- 172.21.15.136:0/1299099025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc815ab40 msgr2=0x7f9cc80ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc815ab40 0x7f9cc80ca6a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009d90 tx=0x7f9cc001f150 comp rx=0 tx=0).stop 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- 172.21.15.136:0/1299099025 shutdown_connections 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc815ab40 0x7f9cc80ca6a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 --2- 172.21.15.136:0/1299099025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- 172.21.15.136:0/1299099025 >> 172.21.15.136:0/1299099025 conn(0x7f9cc8070f70 msgr2=0x7f9cc8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.254+0000 7f9cceb69640 1 -- 172.21.15.136:0/1299099025 shutdown_connections 2024-04-25T11:38:19.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 -- 172.21.15.136:0/1299099025 wait complete. 2024-04-25T11:38:19.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 Processor -- start 2024-04-25T11:38:19.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 -- start start 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 0x7f9cc8144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc815d3f0 con 0x7f9cc80ca290 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8159f10 con 0x7f9cc815ab40 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cceb69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc80c4a60 con 0x7f9cc80b5060 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45400/0 (socket says 172.21.15.136:45400) 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 -- 172.21.15.136:0/2274488563 learned_addr learned my addr 172.21.15.136:0/2274488563 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9ccc905640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9ccd106640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 0x7f9cc8144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 -- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 msgr2=0x7f9cc80b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 -- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 msgr2=0x7f9cc8144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 0x7f9cc8144d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 -- 172.21.15.136:0/2274488563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc81f1300 con 0x7f9cc80ca290 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9ccd106640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 0x7f9cc8144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.258+0000 7f9cc7fff640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c370 tx=0x7f9cbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc017070 con 0x7f9cc80ca290 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc81f15e0 con 0x7f9cc80ca290 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc005cf0 con 0x7f9cc80ca290 2024-04-25T11:38:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc0072f0 con 0x7f9cc80ca290 2024-04-25T11:38:19.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc80b3840 con 0x7f9cc80ca290 2024-04-25T11:38:19.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc81f1800 con 0x7f9cc80ca290 2024-04-25T11:38:19.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9cbc01a070 con 0x7f9cc80ca290 2024-04-25T11:38:19.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 0x7f9cac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:19.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f9cbc08f1a0 con 0x7f9cc80ca290 2024-04-25T11:38:19.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9ccc905640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 0x7f9cac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:19.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.262+0000 7f9ccc905640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 0x7f9cac071940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b930 tx=0x7f9cb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:19.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.266+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cc81f1800 con 0x7f9cc80ca290 2024-04-25T11:38:19.446 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.462+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc806a350 con 0x7f9cc80ca290 2024-04-25T11:38:19.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.462+0000 7f9cc57fa640 1 -- 172.21.15.136:0/2274488563 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6098 (secure 0 0 0) 0x7f9cbc021020 con 0x7f9cc80ca290 2024-04-25T11:38:19.448 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:19.448 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:18.992694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:19.448 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 74 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.462+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 msgr2=0x7f9cac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.462+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 0x7f9cac071940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b930 tx=0x7f9cb4008040 comp rx=0 tx=0).stop 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 msgr2=0x7f9cc8144840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c370 tx=0x7f9cbc00c830 comp rx=0 tx=0).stop 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 shutdown_connections 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cc815ab40 0x7f9cc8144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80ca290 0x7f9cc8144840 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.450 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9cac06f430 0x7f9cac071940 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 --2- 172.21.15.136:0/2274488563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9cc80b5060 0x7f9cc80b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:19.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 >> 172.21.15.136:0/2274488563 conn(0x7f9cc8070f70 msgr2=0x7f9cc80cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:19.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 shutdown_connections 2024-04-25T11:38:19.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:19.466+0000 7f9cceb69640 1 -- 172.21.15.136:0/2274488563 wait complete. 2024-04-25T11:38:19.471 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5408": { "addr": "172.21.15.157:6837/1501623433", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 1501623433, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 1501623433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5408, "incarnation": 74, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:38:18.992694+0000", "qdb_cluster": [ 5385, 5449, 5517, 6246 ], "qdb_leader": 5449, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5408, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:38:21.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:21.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 Processor -- start 2024-04-25T11:38:21.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 -- start start 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f4758152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4758071300 con 0x7f4758158590 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47580c7300 con 0x7f47581589a0 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475fa8b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47580710d0 con 0x7f47580c4750 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475d026640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4758158590 0x7f47580736a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36340/0 (socket says 172.21.15.136:36340) 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475d026640 1 -- 172.21.15.136:0/2075611868 learned_addr learned my addr 172.21.15.136:0/2075611868 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475d827640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f4758152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2828632443 0 0) 0x7f4758071300 con 0x7f4758158590 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4744004640 con 0x7f4758158590 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2188254989 0 0) 0x7f4744004640 con 0x7f4758158590 2024-04-25T11:38:21.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 msgr2=0x7f4758152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f4758152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 >> v1:172.21.15.157:6789/0 conn(0x7f47581589a0 legacy=0x7f4758162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47580c8fa0 con 0x7f4758158590 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f4748001680 con 0x7f4758158590 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 >> v1:172.21.15.136:6789/0 conn(0x7f4758158590 legacy=0x7f47580736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4744004b00 0x7f4744004f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4744005920 0x7f4744005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f474400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4744006680 con 0x7f4744005920 2024-04-25T11:38:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f474400d020 con 0x7f4744004b00 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.650+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f474400d1a0 con 0x7f47580c4750 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4744005920 0x7f4744005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d827640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4744004b00 0x7f4744004f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475e028640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f474400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 -- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 msgr2=0x7f474400cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47580c4750 0x7f474400cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 -- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4744004b00 msgr2=0x7f4744004f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4744004b00 0x7f4744004f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.654+0000 7f475d026640 1 -- 172.21.15.136:0/2075611868 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4744004640 con 0x7f4744005920 2024-04-25T11:38:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475d026640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4744005920 0x7f4744005d30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f474800c210 tx=0x7f474801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15516508d03b271f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748007d40 con 0x7f4744005920 2024-04-25T11:38:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47480a9070 con 0x7f4744005920 2024-04-25T11:38:21.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475c825640 1 -- 172.21.15.136:0/2075611868 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748005d40 con 0x7f4744005920 2024-04-25T11:38:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 -- 172.21.15.136:0/2075611868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4744005920 msgr2=0x7f4744005d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 --2- 172.21.15.136:0/2075611868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4744005920 0x7f4744005d30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f474800c210 tx=0x7f474801cc40 comp rx=0 tx=0).stop 2024-04-25T11:38:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475e028640 1 -- 172.21.15.136:0/2075611868 reap_dead start 2024-04-25T11:38:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 -- 172.21.15.136:0/2075611868 shutdown_connections 2024-04-25T11:38:21.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 -- 172.21.15.136:0/2075611868 >> 172.21.15.136:0/2075611868 conn(0x7f47580c9d50 msgr2=0x7f47580ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:21.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 -- 172.21.15.136:0/2075611868 shutdown_connections 2024-04-25T11:38:21.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.658+0000 7f475fa8b640 1 -- 172.21.15.136:0/2075611868 wait complete. 2024-04-25T11:38:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 Processor -- start 2024-04-25T11:38:21.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- start start 2024-04-25T11:38:21.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4744005920 0x7f4758151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f475815c710 0x7f47581f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4758152d10 con 0x7f475806e390 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47580c95f0 con 0x7f475815c710 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4758073db0 con 0x7f4744005920 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45404/0 (socket says 172.21.15.136:45404) 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 -- 172.21.15.136:0/4257896415 learned_addr learned my addr 172.21.15.136:0/4257896415 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 -- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4744005920 msgr2=0x7f4758151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4744005920 0x7f4758151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 -- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f475815c710 msgr2=0x7f47581f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f475815c710 0x7f47581f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 -- 172.21.15.136:0/4257896415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475814b180 con 0x7f475806e390 2024-04-25T11:38:21.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475e028640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f475806e190 tx=0x7f475000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:21.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750003b90 con 0x7f475806e390 2024-04-25T11:38:21.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4750018070 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475814b460 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47500050d0 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.662+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4758151200 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475806a350 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4750005270 con 0x7f475806e390 2024-04-25T11:38:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f474e7fc640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 0x7f473c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f475008fbf0 con 0x7f475806e390 2024-04-25T11:38:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f475d827640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 0x7f473c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.666+0000 7f475d827640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 0x7f473c071940 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4754009ad0 tx=0x7f47540095f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.670+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f475005c480 con 0x7f475806e390 2024-04-25T11:38:21.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.866+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4758070510 con 0x7f475806e390 2024-04-25T11:38:21.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f474e7fc640 1 -- 172.21.15.136:0/4257896415 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6098 (secure 0 0 0) 0x7f47500018d0 con 0x7f475806e390 2024-04-25T11:38:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:21.855 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:18.992694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:21.855 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 74 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 msgr2=0x7f473c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 0x7f473c071940 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4754009ad0 tx=0x7f47540095f0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 msgr2=0x7f4758146ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f475806e190 tx=0x7f475000fd40 comp rx=0 tx=0).stop 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 shutdown_connections 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f475806e390 0x7f4758146ab0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.857 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f475815c710 0x7f47581f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f473c06f430 0x7f473c071940 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 --2- 172.21.15.136:0/4257896415 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4744005920 0x7f4758151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:21.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 >> 172.21.15.136:0/4257896415 conn(0x7f47580c9d50 msgr2=0x7f47580736a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:21.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 shutdown_connections 2024-04-25T11:38:21.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:21.870+0000 7f475fa8b640 1 -- 172.21.15.136:0/4257896415 wait complete. 2024-04-25T11:38:21.878 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:23.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:24.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 Processor -- start 2024-04-25T11:38:24.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 -- start start 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 0x7fbd280c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd2815d750 con 0x7fbd2815ab40 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd2815da30 con 0x7fbd280ca290 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2edfa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2815dd10 con 0x7fbd280b5060 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2d397640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbd2815ab40 0x7fbd28164780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:36354/0 (socket says 172.21.15.136:36354) 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2d397640 1 -- 172.21.15.136:0/1856004885 learned_addr learned my addr 172.21.15.136:0/1856004885 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2cb96640 1 --2- 172.21.15.136:0/1856004885 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 0x7fbd280c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160391033 0 0) 0x7fbd2815d750 con 0x7fbd2815ab40 2024-04-25T11:38:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd10003530 con 0x7fbd2815ab40 2024-04-25T11:38:24.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2cb96640 1 -- 172.21.15.136:0/1856004885 >> v1:172.21.15.157:6789/0 conn(0x7fbd280ca290 legacy=0x7fbd2815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1386370321 0 0) 0x7fbd10003530 con 0x7fbd2815ab40 2024-04-25T11:38:24.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2cb96640 1 -- 172.21.15.136:0/1856004885 >> v1:172.21.15.136:6789/0 conn(0x7fbd2815ab40 legacy=0x7fbd28164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.062+0000 7fbd2cb96640 1 -- 172.21.15.136:0/1856004885 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2815dff0 con 0x7fbd280b5060 2024-04-25T11:38:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.066+0000 7fbd2cb96640 1 --2- 172.21.15.136:0/1856004885 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 0x7fbd280c4520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd180097f0 tx=0x7fbd1801c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31a1b1d216e7249 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.066+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd180b0070 con 0x7fbd280b5060 2024-04-25T11:38:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.066+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd180ab400 con 0x7fbd280b5060 2024-04-25T11:38:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.066+0000 7fbd277fe640 1 -- 172.21.15.136:0/1856004885 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20001a00 con 0x7fbd280b5060 2024-04-25T11:38:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1856004885 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 msgr2=0x7fbd280c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1856004885 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 0x7fbd280c4520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd180097f0 tx=0x7fbd1801c340 comp rx=0 tx=0).stop 2024-04-25T11:38:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1856004885 shutdown_connections 2024-04-25T11:38:24.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1856004885 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280b5060 0x7fbd280c4520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1856004885 >> 172.21.15.136:0/1856004885 conn(0x7fbd28070f70 msgr2=0x7fbd28068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:24.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1856004885 shutdown_connections 2024-04-25T11:38:24.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1856004885 wait complete. 2024-04-25T11:38:24.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 Processor -- start 2024-04-25T11:38:24.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- start start 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280ca290 0x7fbd281f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd28159990 0x7fbd2814f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2815d3f0 con 0x7fbd2815ab40 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd28159f10 con 0x7fbd28159990 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.070+0000 7fbd2edfa640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbd280c4a60 con 0x7fbd280ca290 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45406/0 (socket says 172.21.15.136:45406) 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 -- 172.21.15.136:0/1748743288 learned_addr learned my addr 172.21.15.136:0/1748743288 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2cb96640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280ca290 0x7fbd281f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:24.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2d397640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd28159990 0x7fbd2814f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 -- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280ca290 msgr2=0x7fbd281f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280ca290 0x7fbd281f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 -- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd28159990 msgr2=0x7fbd2814f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd28159990 0x7fbd2814f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 -- 172.21.15.136:0/1748743288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd281f4790 con 0x7fbd2815ab40 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd27fff640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c00c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c00cd50 con 0x7fbd2815ab40 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c004020 con 0x7fbd2815ab40 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd281f14a0 con 0x7fbd2815ab40 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c004410 con 0x7fbd2815ab40 2024-04-25T11:38:24.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd28146930 con 0x7fbd2815ab40 2024-04-25T11:38:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd2806a350 con 0x7fbd2815ab40 2024-04-25T11:38:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbd1c01b020 con 0x7fbd2815ab40 2024-04-25T11:38:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 0x7fbd14071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fbd1c092490 con 0x7fbd2815ab40 2024-04-25T11:38:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.074+0000 7fbd2cb96640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 0x7fbd14071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:24.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.078+0000 7fbd2cb96640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 0x7fbd14071910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbd180ad000 tx=0x7fbd180a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:24.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.082+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbd1c05e5d0 con 0x7fbd2815ab40 2024-04-25T11:38:24.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:38:24.201 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:38:24.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.286+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd281470f0 con 0x7fbd2815ab40 2024-04-25T11:38:24.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.286+0000 7fbd257fa640 1 -- 172.21.15.136:0/1748743288 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+6098 (secure 0 0 0) 0x7fbd1c0628d0 con 0x7fbd2815ab40 2024-04-25T11:38:24.272 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:24.272 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:18.992694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:24.272 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 74 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 msgr2=0x7fbd14071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 0x7fbd14071910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbd180ad000 tx=0x7fbd180a6000 comp rx=0 tx=0).stop 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 msgr2=0x7fbd28144840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009e50 tx=0x7fbd1c00c430 comp rx=0 tx=0).stop 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 shutdown_connections 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd28159990 0x7fbd2814f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd2815ab40 0x7fbd28144840 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd1406f400 0x7fbd14071910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 --2- 172.21.15.136:0/1748743288 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd280ca290 0x7fbd281f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:24.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 >> 172.21.15.136:0/1748743288 conn(0x7fbd28070f70 msgr2=0x7fbd28150490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:24.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 shutdown_connections 2024-04-25T11:38:24.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:24.290+0000 7fbd2edfa640 1 -- 172.21.15.136:0/1748743288 wait complete. 2024-04-25T11:38:24.295 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:26.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:26.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 Processor -- start 2024-04-25T11:38:26.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 -- start start 2024-04-25T11:38:26.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb6e8156330 0x7fb6e815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e815d010 con 0x7fb6e80cdee0 2024-04-25T11:38:26.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e81602f0 con 0x7fb6e8156330 2024-04-25T11:38:26.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6ed8af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e80ccb80 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6e77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6e67fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb6e80cdee0 0x7fb6e80b5490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:38640/0 (socket says 172.21.15.136:38640) 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6e67fc640 1 -- 172.21.15.136:0/3212367292 learned_addr learned my addr 172.21.15.136:0/3212367292 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6e6ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb6e8156330 0x7fb6e815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.474+0000 7fb6e77fe640 1 --2- 172.21.15.136:0/3212367292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:46882/0 (socket says 172.21.15.136:46882) 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e77fe640 1 -- 172.21.15.136:0/3212367292 >> v2:172.21.15.157:3300/0 conn(0x7fb6e8156330 msgr2=0x7fb6e815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e77fe640 1 --2- 172.21.15.136:0/3212367292 >> v2:172.21.15.157:3300/0 conn(0x7fb6e8156330 0x7fb6e815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e77fe640 1 -- 172.21.15.136:0/3212367292 >> v1:172.21.15.157:6789/0 conn(0x7fb6e80cdee0 legacy=0x7fb6e80b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e77fe640 1 -- 172.21.15.136:0/3212367292 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e8156740 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e77fe640 1 --2- 172.21.15.136:0/3212367292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009d90 tx=0x7fb6e001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f796d0ebc1e6f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:26.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e5ffb640 1 -- 172.21.15.136:0/3212367292 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00b4070 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e5ffb640 1 -- 172.21.15.136:0/3212367292 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e00af600 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6e5ffb640 1 -- 172.21.15.136:0/3212367292 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00af9d0 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 -- 172.21.15.136:0/3212367292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 msgr2=0x7fb6e80ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/3212367292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0009d90 tx=0x7fb6e001f150 comp rx=0 tx=0).stop 2024-04-25T11:38:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 -- 172.21.15.136:0/3212367292 shutdown_connections 2024-04-25T11:38:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/3212367292 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e80b5ba0 0x7fb6e80ce2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/3212367292 >> v2:172.21.15.157:3300/0 conn(0x7fb6e8156330 0x7fb6e815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 -- 172.21.15.136:0/3212367292 >> 172.21.15.136:0/3212367292 conn(0x7fb6e806d6c0 msgr2=0x7fb6e806e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:26.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 -- 172.21.15.136:0/3212367292 shutdown_connections 2024-04-25T11:38:26.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.478+0000 7fb6ed8af640 1 -- 172.21.15.136:0/3212367292 wait complete. 2024-04-25T11:38:26.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 Processor -- start 2024-04-25T11:38:26.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- start start 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 0x7fb6e81f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e8156330 0x7fb6e8151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e815fca0 con 0x7fb6e80cdee0 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e815f9c0 con 0x7fb6e80b5ba0 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e815ff80 con 0x7fb6e8156330 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55506/0 (socket says 172.21.15.136:55506) 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 -- 172.21.15.136:0/743816514 learned_addr learned my addr 172.21.15.136:0/743816514 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e77fe640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e8156330 0x7fb6e8151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e6ffd640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 0x7fb6e81f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 -- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e8156330 msgr2=0x7fb6e8151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e8156330 0x7fb6e8151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 -- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 msgr2=0x7fb6e81f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 0x7fb6e81f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 -- 172.21.15.136:0/743816514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e80cb1c0 con 0x7fb6e80cdee0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e6ffd640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 0x7fb6e81f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6e67fc640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800c340 tx=0x7fb6d800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8017070 con 0x7fb6e80cdee0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d8005cf0 con 0x7fb6e80cdee0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80072f0 con 0x7fb6e80cdee0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e80c9c70 con 0x7fb6e80cdee0 2024-04-25T11:38:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.482+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e80ca230 con 0x7fb6e80cdee0 2024-04-25T11:38:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e806a350 con 0x7fb6e80cdee0 2024-04-25T11:38:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb6d801a070 con 0x7fb6e80cdee0 2024-04-25T11:38:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6cffff640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 0x7fb6d4071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fb6d808eb50 con 0x7fb6e80cdee0 2024-04-25T11:38:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6e6ffd640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 0x7fb6d4071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:26.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.486+0000 7fb6e6ffd640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 0x7fb6d4071990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009b10 tx=0x7fb6dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.490+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb6e815d010 con 0x7fb6e80cdee0 2024-04-25T11:38:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6e80cd9c0 con 0x7fb6e80cdee0 2024-04-25T11:38:26.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6cffff640 1 -- 172.21.15.136:0/743816514 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+6100 (secure 0 0 0) 0x7fb6d805ae00 con 0x7fb6e80cdee0 2024-04-25T11:38:26.680 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:26.680 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:24.373138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:resolve","state_seq":15,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5449,5385]},"id":1}]} 2024-04-25T11:38:26.680 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 75 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 msgr2=0x7fb6d4071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 0x7fb6d4071990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009b10 tx=0x7fb6dc008040 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 msgr2=0x7fb6e80d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800c340 tx=0x7fb6d800c800 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 shutdown_connections 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6d406f480 0x7fb6d4071990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb6e8156330 0x7fb6e8151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb6e80cdee0 0x7fb6e80d0130 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 --2- 172.21.15.136:0/743816514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6e80b5ba0 0x7fb6e81f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 >> 172.21.15.136:0/743816514 conn(0x7fb6e806d6c0 msgr2=0x7fb6e806e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.694+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 shutdown_connections 2024-04-25T11:38:26.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:26.698+0000 7fb6ed8af640 1 -- 172.21.15.136:0/743816514 wait complete. 2024-04-25T11:38:26.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:28.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.878+0000 7facce51f640 1 Processor -- start 2024-04-25T11:38:28.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.878+0000 7facce51f640 1 -- start start 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facce51f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7facc8153190 0x7facc80c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facce51f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facc81594b0 0x7facc8163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facce51f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc80c8090 con 0x7facc80c9570 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facce51f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7facc80c8370 con 0x7facc8153190 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facce51f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7facc80c6aa0 con 0x7facc81594b0 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7facc8153190 0x7facc80c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7facc8153190 0x7facc80c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55516/0 (socket says 172.21.15.136:55516) 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 -- 172.21.15.136:0/895402559 learned_addr learned my addr 172.21.15.136:0/895402559 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:28.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facccabc640 1 --2- 172.21.15.136:0/895402559 >> v2:172.21.15.157:3300/0 conn(0x7facc81594b0 0x7facc8163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:28.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 -- 172.21.15.136:0/895402559 >> v2:172.21.15.157:3300/0 conn(0x7facc81594b0 msgr2=0x7facc8163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:28.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 --2- 172.21.15.136:0/895402559 >> v2:172.21.15.157:3300/0 conn(0x7facc81594b0 0x7facc8163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 -- 172.21.15.136:0/895402559 >> v1:172.21.15.157:6789/0 conn(0x7facc80c9570 legacy=0x7facc8158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:28.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.882+0000 7facc7fff640 1 -- 172.21.15.136:0/895402559 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc80c6d80 con 0x7facc8153190 2024-04-25T11:38:28.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facc7fff640 1 --2- 172.21.15.136:0/895402559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc8153190 0x7facc80c9030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7facc8075070 tx=0x7facb80213a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e0894c832c3beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:28.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facc6ffd640 1 -- 172.21.15.136:0/895402559 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80b6070 con 0x7facc8153190 2024-04-25T11:38:28.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facc6ffd640 1 -- 172.21.15.136:0/895402559 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb80b1600 con 0x7facc8153190 2024-04-25T11:38:28.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facc6ffd640 1 -- 172.21.15.136:0/895402559 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80b19d0 con 0x7facc8153190 2024-04-25T11:38:28.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- 172.21.15.136:0/895402559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc8153190 msgr2=0x7facc80c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:28.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 --2- 172.21.15.136:0/895402559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc8153190 0x7facc80c9030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7facc8075070 tx=0x7facb80213a0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- 172.21.15.136:0/895402559 shutdown_connections 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 --2- 172.21.15.136:0/895402559 >> v2:172.21.15.157:3300/0 conn(0x7facc81594b0 0x7facc8163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 --2- 172.21.15.136:0/895402559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc8153190 0x7facc80c9030 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- 172.21.15.136:0/895402559 >> 172.21.15.136:0/895402559 conn(0x7facc80c9d50 msgr2=0x7facc80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- 172.21.15.136:0/895402559 shutdown_connections 2024-04-25T11:38:28.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- 172.21.15.136:0/895402559 wait complete. 2024-04-25T11:38:28.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 Processor -- start 2024-04-25T11:38:28.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.886+0000 7facce51f640 1 -- start start 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facc8153190 0x7facc81f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facc81594b0 0x7facc8158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7facc80ca870 con 0x7facc80c9570 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facc80c7340 con 0x7facc8153190 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7facc80ca9f0 con 0x7facc81594b0 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:28.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55522/0 (socket says 172.21.15.136:55522) 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 -- 172.21.15.136:0/3336803726 learned_addr learned my addr 172.21.15.136:0/3336803726 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 -- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facc81594b0 msgr2=0x7facc8158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facc81594b0 0x7facc8158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 -- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facc8153190 msgr2=0x7facc81f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facc8153190 0x7facc81f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 -- 172.21.15.136:0/3336803726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc81f5eb0 con 0x7facc80c9570 2024-04-25T11:38:28.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc7fff640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7facb801fa50 tx=0x7facb801fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80c1070 con 0x7facc80c9570 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb80b07b0 con 0x7facc80c9570 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc81f6190 con 0x7facc80c9570 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80b6070 con 0x7facc80c9570 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc81f6750 con 0x7facc80c9570 2024-04-25T11:38:28.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc806a350 con 0x7facc80c9570 2024-04-25T11:38:28.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.890+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7facb80c4070 con 0x7facc80c9570 2024-04-25T11:38:28.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.894+0000 7facc4ff9640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 0x7facb4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:28.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.894+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7facb812d320 con 0x7facc80c9570 2024-04-25T11:38:28.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.894+0000 7facc77fe640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 0x7facb4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:28.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.894+0000 7facc77fe640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 0x7facb4071940 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7facc81f02c0 tx=0x7facbc006bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:28.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:28.898+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facb801af00 con 0x7facc80c9570 2024-04-25T11:38:29.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.086+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facc80c43f0 con 0x7facc80c9570 2024-04-25T11:38:29.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.086+0000 7facc4ff9640 1 -- 172.21.15.136:0/3336803726 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+6100 (secure 0 0 0) 0x7facb80f9cb0 con 0x7facc80c9570 2024-04-25T11:38:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:29.073 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:24.373138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:resolve","state_seq":15,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5449,5385]},"id":1}]} 2024-04-25T11:38:29.073 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 75 2024-04-25T11:38:29.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 msgr2=0x7facb4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:29.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 0x7facb4071940 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7facc81f02c0 tx=0x7facbc006bd0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 msgr2=0x7facc81e2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:29.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7facb801fa50 tx=0x7facb801fbe0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 shutdown_connections 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f430 0x7facb4071940 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facc81594b0 0x7facc8158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facc8153190 0x7facc81f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 --2- 172.21.15.136:0/3336803726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facc80c9570 0x7facc81e2e50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 >> 172.21.15.136:0/3336803726 conn(0x7facc80c9d50 msgr2=0x7facc814da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 shutdown_connections 2024-04-25T11:38:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:29.090+0000 7facce51f640 1 -- 172.21.15.136:0/3336803726 wait complete. 2024-04-25T11:38:29.095 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:31.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:31.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 Processor -- start 2024-04-25T11:38:31.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 -- start start 2024-04-25T11:38:31.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f66f00c8cd0 0x7f66f014f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f00cbad0 con 0x7f66f00ce260 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f66f00c8550 con 0x7f66f00c8cd0 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f7441640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66f00c7c20 con 0x7f66f00ca720 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f51dd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f66f00c8cd0 0x7f66f014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:46916/0 (socket says 172.21.15.136:46916) 2024-04-25T11:38:31.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 -- 172.21.15.136:0/54529372 learned_addr learned my addr 172.21.15.136:0/54529372 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:31.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 -- 172.21.15.136:0/54529372 >> v2:172.21.15.157:3300/0 conn(0x7f66f00c8cd0 msgr2=0x7f66f014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:31.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 --2- 172.21.15.136:0/54529372 >> v2:172.21.15.157:3300/0 conn(0x7f66f00c8cd0 0x7f66f014f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 -- 172.21.15.136:0/54529372 >> v1:172.21.15.157:6789/0 conn(0x7f66f00ce260 legacy=0x7f66f00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.274+0000 7f66f59de640 1 -- 172.21.15.136:0/54529372 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f00c7f00 con 0x7f66f00ca720 2024-04-25T11:38:31.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f59de640 1 --2- 172.21.15.136:0/54529372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66f00c86d0 tx=0x7f66e801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3feee6d43b2e3bce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:31.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66effff640 1 -- 172.21.15.136:0/54529372 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80b5070 con 0x7f66f00ca720 2024-04-25T11:38:31.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66effff640 1 -- 172.21.15.136:0/54529372 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e80a8a30 con 0x7f66f00ca720 2024-04-25T11:38:31.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66effff640 1 -- 172.21.15.136:0/54529372 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80a8e00 con 0x7f66f00ca720 2024-04-25T11:38:31.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 -- 172.21.15.136:0/54529372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 msgr2=0x7f66f00ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 --2- 172.21.15.136:0/54529372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66f00c86d0 tx=0x7f66e801ed30 comp rx=0 tx=0).stop 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 -- 172.21.15.136:0/54529372 shutdown_connections 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 --2- 172.21.15.136:0/54529372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00ca720 0x7f66f00ce670 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 --2- 172.21.15.136:0/54529372 >> v2:172.21.15.157:3300/0 conn(0x7f66f00c8cd0 0x7f66f014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 -- 172.21.15.136:0/54529372 >> 172.21.15.136:0/54529372 conn(0x7f66f006f9d0 msgr2=0x7f66f006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 -- 172.21.15.136:0/54529372 shutdown_connections 2024-04-25T11:38:31.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.278+0000 7f66f7441640 1 -- 172.21.15.136:0/54529372 wait complete. 2024-04-25T11:38:31.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 Processor -- start 2024-04-25T11:38:31.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- start start 2024-04-25T11:38:31.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00c8cd0 0x7f66f0149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f00ce260 0x7f66f0145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00cb720 con 0x7f66f00ca720 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66f014fbc0 con 0x7f66f00ce260 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66f014a270 con 0x7f66f00c8cd0 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55530/0 (socket says 172.21.15.136:55530) 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 -- 172.21.15.136:0/259154717 learned_addr learned my addr 172.21.15.136:0/259154717 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f51dd640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00c8cd0 0x7f66f0149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f59de640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f00ce260 0x7f66f0145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 -- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00c8cd0 msgr2=0x7f66f0149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00c8cd0 0x7f66f0149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 -- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f00ce260 msgr2=0x7f66f0145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f00ce260 0x7f66f0145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 -- 172.21.15.136:0/259154717 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f0158a80 con 0x7f66f00ca720 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f49dc640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c310 tx=0x7f66e400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e40052d0 con 0x7f66f00ca720 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4005470 con 0x7f66f00ca720 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f014d480 con 0x7f66f00ca720 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4007260 con 0x7f66f00ca720 2024-04-25T11:38:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.282+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f014c890 con 0x7f66f00ca720 2024-04-25T11:38:31.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f006a350 con 0x7f66f00ca720 2024-04-25T11:38:31.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f66e4007400 con 0x7f66f00ca720 2024-04-25T11:38:31.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66edffb640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 0x7f66dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:31.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f66e408ed30 con 0x7f66f00ca720 2024-04-25T11:38:31.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66f51dd640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 0x7f66dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:31.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.286+0000 7f66f51dd640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 0x7f66dc071940 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f66e0011340 tx=0x7f66e000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:31.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.290+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f66e405af10 con 0x7f66f00ca720 2024-04-25T11:38:31.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.486+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66f01599e0 con 0x7f66f00ca720 2024-04-25T11:38:31.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.486+0000 7f66edffb640 1 -- 172.21.15.136:0/259154717 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6102 (secure 0 0 0) 0x7f66e405f210 con 0x7f66f00ca720 2024-04-25T11:38:31.473 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:31.473 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:30.352654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:31.474 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 76 2024-04-25T11:38:31.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 msgr2=0x7f66dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 0x7f66dc071940 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f66e0011340 tx=0x7f66e000b040 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 msgr2=0x7f66f0158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c310 tx=0x7f66e400c7d0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 shutdown_connections 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66f00ce260 0x7f66f0145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66f00ca720 0x7f66f0158030 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f66dc06f430 0x7f66dc071940 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 --2- 172.21.15.136:0/259154717 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66f00c8cd0 0x7f66f0149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 >> 172.21.15.136:0/259154717 conn(0x7f66f006f9d0 msgr2=0x7f66f0148cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 shutdown_connections 2024-04-25T11:38:31.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:31.490+0000 7f66f7441640 1 -- 172.21.15.136:0/259154717 wait complete. 2024-04-25T11:38:31.497 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 76, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5385": { "addr": "172.21.15.136:6833/906354453", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 906354453, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 906354453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5385, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 40 }, "gid_5408": { "addr": "172.21.15.157:6837/1501623433", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 1501623433, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 1501623433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5408, "incarnation": 74, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 17 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:38:30.352654+0000", "qdb_cluster": [ 5385, 5449, 5517, 6246 ], "qdb_leader": 5449, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5408, "mds_1": 5449, "mds_2": 5517, "mds_3": 5385, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:38:33.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:33.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 Processor -- start 2024-04-25T11:38:33.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 -- start start 2024-04-25T11:38:33.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b000c66e0 0x7f0b00156380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b000cdee0 0x7f0b000ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b000c6470 con 0x7f0b0015cb70 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0b000cd390 con 0x7f0b000c66e0 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0638f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b0015c6e0 con 0x7f0b000cdee0 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b000c66e0 0x7f0b00156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0aff7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0b0015cb70 0x7f0b0015f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:45434/0 (socket says 172.21.15.136:45434) 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0aff7fe640 1 -- 172.21.15.136:0/3533931429 learned_addr learned my addr 172.21.15.136:0/3533931429 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0b0492c640 1 --2- 172.21.15.136:0/3533931429 >> v2:172.21.15.157:3300/0 conn(0x7f0b000cdee0 0x7f0b000ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afffff640 1 -- 172.21.15.136:0/3533931429 >> v2:172.21.15.157:3300/0 conn(0x7f0b000cdee0 msgr2=0x7f0b000ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afffff640 1 --2- 172.21.15.136:0/3533931429 >> v2:172.21.15.157:3300/0 conn(0x7f0b000cdee0 0x7f0b000ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afffff640 1 -- 172.21.15.136:0/3533931429 >> v1:172.21.15.136:6789/0 conn(0x7f0b0015cb70 legacy=0x7f0b0015f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afeffd640 1 -- 172.21.15.136:0/3533931429 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967748643 0 0) 0x7f0b000c6470 con 0x7f0b0015cb70 2024-04-25T11:38:33.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.674+0000 7f0afffff640 1 -- 172.21.15.136:0/3533931429 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0015c0b0 con 0x7f0b000c66e0 2024-04-25T11:38:33.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0afffff640 1 --2- 172.21.15.136:0/3533931429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b000c66e0 0x7f0b00156380 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0b000c8820 tx=0x7f0af001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d1b7ca3fac287b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:33.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0afeffd640 1 -- 172.21.15.136:0/3533931429 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00b0070 con 0x7f0b000c66e0 2024-04-25T11:38:33.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0afeffd640 1 -- 172.21.15.136:0/3533931429 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af00ab600 con 0x7f0b000c66e0 2024-04-25T11:38:33.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0afeffd640 1 -- 172.21.15.136:0/3533931429 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00ab9d0 con 0x7f0b000c66e0 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 -- 172.21.15.136:0/3533931429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b000c66e0 msgr2=0x7f0b00156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3533931429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b000c66e0 0x7f0b00156380 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0b000c8820 tx=0x7f0af001c480 comp rx=0 tx=0).stop 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 -- 172.21.15.136:0/3533931429 shutdown_connections 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3533931429 >> v2:172.21.15.157:3300/0 conn(0x7f0b000cdee0 0x7f0b000ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3533931429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b000c66e0 0x7f0b00156380 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 -- 172.21.15.136:0/3533931429 >> 172.21.15.136:0/3533931429 conn(0x7f0b0006d6c0 msgr2=0x7f0b0006e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 -- 172.21.15.136:0/3533931429 shutdown_connections 2024-04-25T11:38:33.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.678+0000 7f0b0638f640 1 -- 172.21.15.136:0/3533931429 wait complete. 2024-04-25T11:38:33.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 Processor -- start 2024-04-25T11:38:33.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 -- start start 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 0x7f0b001f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b000cdee0 0x7f0b0015dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0015cf80 con 0x7f0b0015cb70 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0015c370 con 0x7f0b000c66e0 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0638f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b00169250 con 0x7f0b000cdee0 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55544/0 (socket says 172.21.15.136:55544) 2024-04-25T11:38:33.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 -- 172.21.15.136:0/3105743858 learned_addr learned my addr 172.21.15.136:0/3105743858 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0afffff640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 0x7f0b001f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0aff7fe640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b000cdee0 0x7f0b0015dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 -- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b000cdee0 msgr2=0x7f0b0015dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b000cdee0 0x7f0b0015dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 -- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 msgr2=0x7f0b001f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 0x7f0b001f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 -- 172.21.15.136:0/3105743858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b001f59a0 con 0x7f0b0015cb70 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0afffff640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 0x7f0b001f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0b0492c640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006e90 tx=0x7f0af800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.682+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80125f0 con 0x7f0b0015cb70 2024-04-25T11:38:33.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af8019070 con 0x7f0b0015cb70 2024-04-25T11:38:33.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af80129d0 con 0x7f0b0015cb70 2024-04-25T11:38:33.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b000cac20 con 0x7f0b0015cb70 2024-04-25T11:38:33.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b001f5c60 con 0x7f0b0015cb70 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b000ca6a0 con 0x7f0b0015cb70 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0af8012b70 con 0x7f0b0015cb70 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afcff9640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 0x7f0ae0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f0af8090930 con 0x7f0b0015cb70 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afffff640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 0x7f0ae0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:33.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.686+0000 7f0afffff640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 0x7f0ae0071940 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0af00ad000 tx=0x7f0af0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:33.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.690+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0b000ca6a0 con 0x7f0b0015cb70 2024-04-25T11:38:33.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.890+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b0006a350 con 0x7f0b0015cb70 2024-04-25T11:38:33.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.890+0000 7f0afcff9640 1 -- 172.21.15.136:0/3105743858 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6102 (secure 0 0 0) 0x7f0af805d0c0 con 0x7f0b0015cb70 2024-04-25T11:38:33.876 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:33.876 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:30.352654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:33.876 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 76 2024-04-25T11:38:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 msgr2=0x7f0ae0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 0x7f0ae0071940 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0af00ad000 tx=0x7f0af0008040 comp rx=0 tx=0).stop 2024-04-25T11:38:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 msgr2=0x7f0b000c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0af8006e90 tx=0x7f0af800d9e0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 shutdown_connections 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0015cb70 0x7f0b000c8950 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0ae006f430 0x7f0ae0071940 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b000cdee0 0x7f0b0015dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 --2- 172.21.15.136:0/3105743858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b000c66e0 0x7f0b001f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 >> 172.21.15.136:0/3105743858 conn(0x7f0b0006d6c0 msgr2=0x7f0b001507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 shutdown_connections 2024-04-25T11:38:33.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:33.894+0000 7f0b0638f640 1 -- 172.21.15.136:0/3105743858 wait complete. 2024-04-25T11:38:33.899 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:35.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:36.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.078+0000 7fca7dbd8640 1 Processor -- start 2024-04-25T11:38:36.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.078+0000 7fca7dbd8640 1 -- start start 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca7dbd8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fca780ca010 0x7fca7806cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca7dbd8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 0x7fca7815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca7dbd8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca780c8a20 con 0x7fca780c9b50 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca7dbd8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fca7806c670 con 0x7fca780ca010 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca7dbd8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fca780c8d00 con 0x7fca7806d040 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca777fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fca780ca010 0x7fca7806cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca76ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fca780c9b50 0x7fca7815bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44962/0 (socket says 172.21.15.136:44962) 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca76ffd640 1 -- 172.21.15.136:0/2053274357 learned_addr learned my addr 172.21.15.136:0/2053274357 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca77fff640 1 --2- 172.21.15.136:0/2053274357 >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 0x7fca7815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca777fe640 1 -- 172.21.15.136:0/2053274357 >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 msgr2=0x7fca7815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca777fe640 1 --2- 172.21.15.136:0/2053274357 >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 0x7fca7815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca777fe640 1 -- 172.21.15.136:0/2053274357 >> v1:172.21.15.136:6789/0 conn(0x7fca780c9b50 legacy=0x7fca7815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca767fc640 1 -- 172.21.15.136:0/2053274357 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264679266 0 0) 0x7fca780c8a20 con 0x7fca780c9b50 2024-04-25T11:38:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca777fe640 1 -- 172.21.15.136:0/2053274357 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca78074d10 con 0x7fca780ca010 2024-04-25T11:38:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.082+0000 7fca77fff640 1 --2- 172.21.15.136:0/2053274357 >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 0x7fca7815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:36.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca777fe640 1 --2- 172.21.15.136:0/2053274357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780ca010 0x7fca7806cb00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fca78070970 tx=0x7fca6801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8290fffff0ec0fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca767fc640 1 -- 172.21.15.136:0/2053274357 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680b0070 con 0x7fca780ca010 2024-04-25T11:38:36.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca767fc640 1 -- 172.21.15.136:0/2053274357 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca680ab600 con 0x7fca780ca010 2024-04-25T11:38:36.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca767fc640 1 -- 172.21.15.136:0/2053274357 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca680ab9d0 con 0x7fca780ca010 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- 172.21.15.136:0/2053274357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780ca010 msgr2=0x7fca7806cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/2053274357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780ca010 0x7fca7806cb00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fca78070970 tx=0x7fca6801c480 comp rx=0 tx=0).stop 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- 172.21.15.136:0/2053274357 shutdown_connections 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/2053274357 >> v2:172.21.15.157:3300/0 conn(0x7fca7806d040 0x7fca7815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/2053274357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780ca010 0x7fca7806cb00 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- 172.21.15.136:0/2053274357 >> 172.21.15.136:0/2053274357 conn(0x7fca780ca5a0 msgr2=0x7fca780ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- 172.21.15.136:0/2053274357 shutdown_connections 2024-04-25T11:38:36.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- 172.21.15.136:0/2053274357 wait complete. 2024-04-25T11:38:36.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 Processor -- start 2024-04-25T11:38:36.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.086+0000 7fca7dbd8640 1 -- start start 2024-04-25T11:38:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca7806d040 0x7fca78157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 0x7fca78149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fca7815cdb0 con 0x7fca780c9b50 2024-04-25T11:38:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca7815cc30 con 0x7fca780ca010 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca78169140 con 0x7fca7806d040 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36248/0 (socket says 172.21.15.136:36248) 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 -- 172.21.15.136:0/3222225621 learned_addr learned my addr 172.21.15.136:0/3222225621 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca777fe640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca7806d040 0x7fca78157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca77fff640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 0x7fca78149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 -- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca7806d040 msgr2=0x7fca78157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca7806d040 0x7fca78157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 -- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 msgr2=0x7fca78149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 0x7fca78149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 -- 172.21.15.136:0/3222225621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca781583f0 con 0x7fca780c9b50 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca77fff640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 0x7fca78149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca76ffd640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c490 tx=0x7fca6000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60017070 con 0x7fca780c9b50 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca60005cf0 con 0x7fca780c9b50 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca781f5740 con 0x7fca780c9b50 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca600072f0 con 0x7fca780c9b50 2024-04-25T11:38:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca781f5d00 con 0x7fca780c9b50 2024-04-25T11:38:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca781f54b0 con 0x7fca780c9b50 2024-04-25T11:38:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fca6001a070 con 0x7fca780c9b50 2024-04-25T11:38:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 0x7fca58071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.090+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fca6008ed70 con 0x7fca780c9b50 2024-04-25T11:38:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.094+0000 7fca777fe640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 0x7fca58071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:36.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.094+0000 7fca777fe640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 0x7fca58071910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fca680ad000 tx=0x7fca680a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:36.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.098+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fca6005b590 con 0x7fca780c9b50 2024-04-25T11:38:36.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.294+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca7806a350 con 0x7fca780c9b50 2024-04-25T11:38:36.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.294+0000 7fca6ffff640 1 -- 172.21.15.136:0/3222225621 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6102 (secure 0 0 0) 0x7fca6005ae90 con 0x7fca780c9b50 2024-04-25T11:38:36.282 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:36.282 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:30.352654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5385,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:36.282 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 76 2024-04-25T11:38:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 msgr2=0x7fca58071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 0x7fca58071910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fca680ad000 tx=0x7fca680a6000 comp rx=0 tx=0).stop 2024-04-25T11:38:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 msgr2=0x7fca78156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:36.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c490 tx=0x7fca6000c950 comp rx=0 tx=0).stop 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 shutdown_connections 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca780ca010 0x7fca78149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca780c9b50 0x7fca78156ee0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca5806f400 0x7fca58071910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 --2- 172.21.15.136:0/3222225621 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca7806d040 0x7fca78157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 >> 172.21.15.136:0/3222225621 conn(0x7fca780ca5a0 msgr2=0x7fca780cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 shutdown_connections 2024-04-25T11:38:36.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:36.298+0000 7fca7dbd8640 1 -- 172.21.15.136:0/3222225621 wait complete. 2024-04-25T11:38:36.306 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:38.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:38.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.482+0000 7fafc4a59640 1 Processor -- start 2024-04-25T11:38:38.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.482+0000 7fafc4a59640 1 -- start start 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafc4a59640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fafc0143170 0x7fafc015d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafc4a59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc00c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafc4a59640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc0143580 con 0x7fafc0070c80 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafc4a59640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fafc015dd20 con 0x7fafc0143170 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafc4a59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fafc015e000 con 0x7fafc0070870 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbe59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fafc0143170 0x7fafc015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbed9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fafc0070c80 0x7fafc0159e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44964/0 (socket says 172.21.15.136:44964) 2024-04-25T11:38:38.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbed9d640 1 -- 172.21.15.136:0/1368128799 learned_addr learned my addr 172.21.15.136:0/1368128799 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbdd9b640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc00c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbe59c640 1 -- 172.21.15.136:0/1368128799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 msgr2=0x7fafc00c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbe59c640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc00c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbe59c640 1 -- 172.21.15.136:0/1368128799 >> v1:172.21.15.136:6789/0 conn(0x7fafc0070c80 legacy=0x7fafc0159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.486+0000 7fafbe59c640 1 -- 172.21.15.136:0/1368128799 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc006fd90 con 0x7fafc0143170 2024-04-25T11:38:38.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafbe59c640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0143170 0x7fafc015d400 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fafb00097c0 tx=0x7fafb001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28204ccf20a4cd4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafbd59a640 1 -- 172.21.15.136:0/1368128799 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb00b5070 con 0x7fafc0143170 2024-04-25T11:38:38.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafbd59a640 1 -- 172.21.15.136:0/1368128799 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb00a8990 con 0x7fafc0143170 2024-04-25T11:38:38.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafbd59a640 1 -- 172.21.15.136:0/1368128799 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb00a8d60 con 0x7fafc0143170 2024-04-25T11:38:38.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- 172.21.15.136:0/1368128799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0143170 msgr2=0x7fafc015d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0143170 0x7fafc015d400 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fafb00097c0 tx=0x7fafb001ec90 comp rx=0 tx=0).stop 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- 172.21.15.136:0/1368128799 shutdown_connections 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc00c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 --2- 172.21.15.136:0/1368128799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0143170 0x7fafc015d400 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- 172.21.15.136:0/1368128799 >> 172.21.15.136:0/1368128799 conn(0x7fafc00c9d50 msgr2=0x7fafc00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- 172.21.15.136:0/1368128799 shutdown_connections 2024-04-25T11:38:38.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- 172.21.15.136:0/1368128799 wait complete. 2024-04-25T11:38:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 Processor -- start 2024-04-25T11:38:38.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.490+0000 7fafc4a59640 1 -- start start 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc01f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 0x7fafc0139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fafc015e2e0 con 0x7fafc0070c80 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0070350 con 0x7fafc0143170 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fafc00712a0 con 0x7fafc0070870 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36262/0 (socket says 172.21.15.136:36262) 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 -- 172.21.15.136:0/3042223332 learned_addr learned my addr 172.21.15.136:0/3042223332 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbed9d640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 0x7fafc0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbe59c640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc01f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 -- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 msgr2=0x7fafc01f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc01f2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 -- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 msgr2=0x7fafc0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 0x7fafc0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 -- 172.21.15.136:0/3042223332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc01f3150 con 0x7fafc0070c80 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbed9d640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 0x7fafc0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafbdd9b640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fafb400d7d0 tx=0x7fafb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb400b660 con 0x7fafc0070c80 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb4019070 con 0x7fafc0070c80 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc00c4bb0 con 0x7fafc0070c80 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb400ba20 con 0x7fafc0070c80 2024-04-25T11:38:38.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc00c60d0 con 0x7fafc0070c80 2024-04-25T11:38:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc006a350 con 0x7fafc0070c80 2024-04-25T11:38:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fafb400bbc0 con 0x7fafc0070c80 2024-04-25T11:38:38.482 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 0x7fafa0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.494+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fafb4090770 con 0x7fafc0070c80 2024-04-25T11:38:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.498+0000 7fafbe59c640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 0x7fafa0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.498+0000 7fafbe59c640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 0x7fafa0071940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fafb00b2000 tx=0x7fafb00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:38.485 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.498+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fafb405d070 con 0x7fafc0070c80 2024-04-25T11:38:38.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.706+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafc0073540 con 0x7fafc0070c80 2024-04-25T11:38:38.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.706+0000 7fafaeffd640 1 -- 172.21.15.136:0/3042223332 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6099 (secure 0 0 0) 0x7fafb405c970 con 0x7fafc0070c80 2024-04-25T11:38:38.693 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:38.693 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:36.160730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5449,5385]},"id":1}]} 2024-04-25T11:38:38.693 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 77 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 msgr2=0x7fafa0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 0x7fafa0071940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fafb00b2000 tx=0x7fafb00aa070 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 msgr2=0x7fafc0138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fafb400d7d0 tx=0x7fafb400dc90 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 shutdown_connections 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafc0143170 0x7fafc0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafc0070c80 0x7fafc0138ac0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fafa006f430 0x7fafa0071940 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 --2- 172.21.15.136:0/3042223332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafc0070870 0x7fafc01f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:38.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 >> 172.21.15.136:0/3042223332 conn(0x7fafc00c9d50 msgr2=0x7fafc0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:38.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 shutdown_connections 2024-04-25T11:38:38.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:38.710+0000 7fafc4a59640 1 -- 172.21.15.136:0/3042223332 wait complete. 2024-04-25T11:38:38.716 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:40.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:40.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 Processor -- start 2024-04-25T11:38:40.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 -- start start 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe2e8156750 0x7fe2e80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80cfba0 0x7fe2e806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e815c410 con 0x7fe2e80ca7d0 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e815c6b0 con 0x7fe2e8156750 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ef216640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e815c990 con 0x7fe2e80cfba0 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe2e8156750 0x7fe2e80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe2e8156750 0x7fe2e80ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36274/0 (socket says 172.21.15.136:36274) 2024-04-25T11:38:40.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 -- 172.21.15.136:0/112306351 learned_addr learned my addr 172.21.15.136:0/112306351 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:40.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ed7b3640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80cfba0 0x7fe2e806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 -- 172.21.15.136:0/112306351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80cfba0 msgr2=0x7fe2e806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:40.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80cfba0 0x7fe2e806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:40.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 -- 172.21.15.136:0/112306351 >> v1:172.21.15.157:6789/0 conn(0x7fe2e80ca7d0 legacy=0x7fe2e80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:40.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.894+0000 7fe2ecfb2640 1 -- 172.21.15.136:0/112306351 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e815cc70 con 0x7fe2e8156750 2024-04-25T11:38:40.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ecfb2640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e8156750 0x7fe2e80ca290 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009870 tx=0x7fe2dc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f75afab71123f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:40.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2e77fe640 1 -- 172.21.15.136:0/112306351 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0b0070 con 0x7fe2e8156750 2024-04-25T11:38:40.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2e77fe640 1 -- 172.21.15.136:0/112306351 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc0ab600 con 0x7fe2e8156750 2024-04-25T11:38:40.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2e77fe640 1 -- 172.21.15.136:0/112306351 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0ab9d0 con 0x7fe2e8156750 2024-04-25T11:38:40.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 -- 172.21.15.136:0/112306351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e8156750 msgr2=0x7fe2e80ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:40.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e8156750 0x7fe2e80ca290 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009870 tx=0x7fe2dc01c520 comp rx=0 tx=0).stop 2024-04-25T11:38:40.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 -- 172.21.15.136:0/112306351 shutdown_connections 2024-04-25T11:38:40.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80cfba0 0x7fe2e806e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:40.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 --2- 172.21.15.136:0/112306351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e8156750 0x7fe2e80ca290 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:40.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 -- 172.21.15.136:0/112306351 >> 172.21.15.136:0/112306351 conn(0x7fe2e8070f70 msgr2=0x7fe2e8068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:40.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 -- 172.21.15.136:0/112306351 shutdown_connections 2024-04-25T11:38:40.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.898+0000 7fe2ef216640 1 -- 172.21.15.136:0/112306351 wait complete. 2024-04-25T11:38:40.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 Processor -- start 2024-04-25T11:38:40.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- start start 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80ca7d0 0x7fe2e815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 0x7fe2e80b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81692c0 con 0x7fe2e80cf620 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e806dab0 con 0x7fe2e80cfba0 2024-04-25T11:38:40.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e8169440 con 0x7fe2e80ca7d0 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36278/0 (socket says 172.21.15.136:36278) 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 -- 172.21.15.136:0/44437562 learned_addr learned my addr 172.21.15.136:0/44437562 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2e7fff640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 0x7fe2e80b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ecfb2640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80ca7d0 0x7fe2e815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 -- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80ca7d0 msgr2=0x7fe2e815d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80ca7d0 0x7fe2e815d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 -- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 msgr2=0x7fe2e80b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 0x7fe2e80b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 -- 172.21.15.136:0/44437562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e80b47f0 con 0x7fe2e80cf620 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2e7fff640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 0x7fe2e80b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ed7b3640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000ccb0 tx=0x7fe2e000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000ce60 con 0x7fe2e80cf620 2024-04-25T11:38:40.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e00039a0 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e81506e0 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0003db0 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e80b37d0 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.902+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e806a350 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.906+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe2e0003fb0 con 0x7fe2e80cf620 2024-04-25T11:38:40.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.906+0000 7fe2e57fa640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 0x7fe2d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:40.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.906+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fe2e008fe00 con 0x7fe2e80cf620 2024-04-25T11:38:40.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.906+0000 7fe2ecfb2640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 0x7fe2d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:40.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.906+0000 7fe2ecfb2640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 0x7fe2d4071910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0ad000 tx=0x7fe2dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:40.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:40.910+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe2e0014030 con 0x7fe2e80cf620 2024-04-25T11:38:41.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.106+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2e81f9640 con 0x7fe2e80cf620 2024-04-25T11:38:41.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2e57fa640 1 -- 172.21.15.136:0/44437562 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6099 (secure 0 0 0) 0x7fe2e005c620 con 0x7fe2e80cf620 2024-04-25T11:38:41.095 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:41.095 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:36.160730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5449,5385]},"id":1}]} 2024-04-25T11:38:41.095 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 77 2024-04-25T11:38:41.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 msgr2=0x7fe2d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:41.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 0x7fe2d4071910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0ad000 tx=0x7fe2dc008040 comp rx=0 tx=0).stop 2024-04-25T11:38:41.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 msgr2=0x7fe2e8151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000ccb0 tx=0x7fe2e000a5c0 comp rx=0 tx=0).stop 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 shutdown_connections 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e80cf620 0x7fe2e8151f80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe2e80cfba0 0x7fe2e80b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe2d406f400 0x7fe2d4071910 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 --2- 172.21.15.136:0/44437562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe2e80ca7d0 0x7fe2e815d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 >> 172.21.15.136:0/44437562 conn(0x7fe2e8070f70 msgr2=0x7fe2e80cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 shutdown_connections 2024-04-25T11:38:41.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:41.110+0000 7fe2ef216640 1 -- 172.21.15.136:0/44437562 wait complete. 2024-04-25T11:38:41.118 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:38:43.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:43.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.294+0000 7f974807b640 1 Processor -- start 2024-04-25T11:38:43.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.294+0000 7f974807b640 1 -- start start 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f974807b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9740158da0 0x7f97401591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f974807b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 0x7f9740163a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f974807b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97401596f0 con 0x7f9740153190 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f974807b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f97400c8370 con 0x7f9740158da0 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f974807b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97400c8090 con 0x7f9740161650 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9746618640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 0x7f9740163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745e17640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9740153190 0x7f97400c9030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:37430/0 (socket says 172.21.15.136:37430) 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745e17640 1 -- 172.21.15.136:0/2056595058 learned_addr learned my addr 172.21.15.136:0/2056595058 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:38:43.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745616640 1 --2- 172.21.15.136:0/2056595058 >> v2:172.21.15.157:3300/0 conn(0x7f9740158da0 0x7f97401591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745616640 1 -- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 msgr2=0x7f9740163a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:43.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745616640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 0x7f9740163a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745616640 1 -- 172.21.15.136:0/2056595058 >> v1:172.21.15.157:6789/0 conn(0x7f9740153190 legacy=0x7f97400c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9745616640 1 -- 172.21.15.136:0/2056595058 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97400c6aa0 con 0x7f9740158da0 2024-04-25T11:38:43.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.298+0000 7f9746618640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 0x7f9740163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:38:43.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f9745616640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740158da0 0x7f97401591b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f973c0097c0 tx=0x7f973c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b155d8bc30d127f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:43.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f9744e15640 1 -- 172.21.15.136:0/2056595058 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c0b5070 con 0x7f9740158da0 2024-04-25T11:38:43.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f9744e15640 1 -- 172.21.15.136:0/2056595058 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f973c0a8990 con 0x7f9740158da0 2024-04-25T11:38:43.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f9744e15640 1 -- 172.21.15.136:0/2056595058 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c0a8d60 con 0x7f9740158da0 2024-04-25T11:38:43.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 -- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740158da0 msgr2=0x7f97401591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740158da0 0x7f97401591b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f973c0097c0 tx=0x7f973c01ec90 comp rx=0 tx=0).stop 2024-04-25T11:38:43.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 -- 172.21.15.136:0/2056595058 shutdown_connections 2024-04-25T11:38:43.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740161650 0x7f9740163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 --2- 172.21.15.136:0/2056595058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740158da0 0x7f97401591b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 -- 172.21.15.136:0/2056595058 >> 172.21.15.136:0/2056595058 conn(0x7f97400c9d50 msgr2=0x7f97400ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:43.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 -- 172.21.15.136:0/2056595058 shutdown_connections 2024-04-25T11:38:43.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.302+0000 7f974807b640 1 -- 172.21.15.136:0/2056595058 wait complete. 2024-04-25T11:38:43.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 Processor -- start 2024-04-25T11:38:43.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- start start 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 0x7f97401f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 0x7f974006fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f97400c7340 con 0x7f9740158da0 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97400c7060 con 0x7f9740161650 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97400ca870 con 0x7f9740153190 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36288/0 (socket says 172.21.15.136:36288) 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 -- 172.21.15.136:0/339274682 learned_addr learned my addr 172.21.15.136:0/339274682 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9746618640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 0x7f974006fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745e17640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 0x7f97401f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 -- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 msgr2=0x7f97401f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 0x7f97401f5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 -- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 msgr2=0x7f974006fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 0x7f974006fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 -- 172.21.15.136:0/339274682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f974014dd30 con 0x7f9740158da0 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9746618640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 0x7f974006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745e17640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 0x7f97401f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f9745616640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f973c0b3040 tx=0x7f973c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c0bf070 con 0x7f9740158da0 2024-04-25T11:38:43.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f974014e010 con 0x7f9740158da0 2024-04-25T11:38:43.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f973c0b0660 con 0x7f9740158da0 2024-04-25T11:38:43.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c0b5040 con 0x7f9740158da0 2024-04-25T11:38:43.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97401f6350 con 0x7f9740158da0 2024-04-25T11:38:43.295 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.306+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f974006a350 con 0x7f9740158da0 2024-04-25T11:38:43.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.310+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f973c0b0a50 con 0x7f9740158da0 2024-04-25T11:38:43.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.310+0000 7f97327fc640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 0x7f9724071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:43.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.310+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f973c12c730 con 0x7f9740158da0 2024-04-25T11:38:43.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.310+0000 7f9745e17640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 0x7f9724071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:43.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.310+0000 7f9745e17640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 0x7f9724071910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9734005890 tx=0x7f9734009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:43.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.314+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f973c0c8020 con 0x7f9740158da0 2024-04-25T11:38:43.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.510+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97400ca4e0 con 0x7f9740158da0 2024-04-25T11:38:43.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f97327fc640 1 -- 172.21.15.136:0/339274682 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+6104 (secure 0 0 0) 0x7f973c0187f0 con 0x7f9740158da0 2024-04-25T11:38:43.498 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:43.498 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5385,5408,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:43.499 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 78 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 msgr2=0x7f9724071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 0x7f9724071910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9734005890 tx=0x7f9734009040 comp rx=0 tx=0).stop 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 msgr2=0x7f9740145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f973c0b3040 tx=0x7f973c01ba30 comp rx=0 tx=0).stop 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 shutdown_connections 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9740161650 0x7f974006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9740158da0 0x7f9740145f10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f972406f400 0x7f9724071910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 --2- 172.21.15.136:0/339274682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9740153190 0x7f97401f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:43.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 >> 172.21.15.136:0/339274682 conn(0x7f97400c9d50 msgr2=0x7f9740148990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:43.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 shutdown_connections 2024-04-25T11:38:43.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:43.514+0000 7f974807b640 1 -- 172.21.15.136:0/339274682 wait complete. 2024-04-25T11:38:43.523 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5356 2024-04-25T11:38:43.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.d 2024-04-25T11:38:44.523 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-04-25T11:38:44.524 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T11:38:44.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T11:38:44.527 INFO:tasks.ceph.mds.d:Started 2024-04-25T11:38:44.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:44.633 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-04-25T11:38:44.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.702+0000 7fb54200d640 1 Processor -- start 2024-04-25T11:38:44.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.702+0000 7fb54200d640 1 -- start start 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb54200d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c143170 0x7fb53c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb54200d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb53c070870 0x7fb53c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb54200d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb53c143580 con 0x7fb53c070c80 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb54200d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb53c15dd20 con 0x7fb53c070870 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb54200d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb53c15e000 con 0x7fb53c143170 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb53c070870 0x7fb53c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb53c070870 0x7fb53c0c6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:32772/0 (socket says 172.21.15.136:32772) 2024-04-25T11:38:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 -- 172.21.15.136:0/1026902938 learned_addr learned my addr 172.21.15.136:0/1026902938 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53b7fe640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c143170 0x7fb53c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 -- 172.21.15.136:0/1026902938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c143170 msgr2=0x7fb53c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:38:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c143170 0x7fb53c15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 -- 172.21.15.136:0/1026902938 >> v1:172.21.15.157:6789/0 conn(0x7fb53c070c80 legacy=0x7fb53c159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.706+0000 7fb53affd640 1 -- 172.21.15.136:0/1026902938 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c06fd90 con 0x7fb53c070870 2024-04-25T11:38:44.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb53affd640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c070870 0x7fb53c0c6960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009870 tx=0x7fb52c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e3e52b5e26ddb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:44.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb53a7fc640 1 -- 172.21.15.136:0/1026902938 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0b5070 con 0x7fb53c070870 2024-04-25T11:38:44.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb53a7fc640 1 -- 172.21.15.136:0/1026902938 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c0a8a50 con 0x7fb53c070870 2024-04-25T11:38:44.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb53a7fc640 1 -- 172.21.15.136:0/1026902938 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0a8e20 con 0x7fb53c070870 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- 172.21.15.136:0/1026902938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c070870 msgr2=0x7fb53c0c6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c070870 0x7fb53c0c6960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009870 tx=0x7fb52c01ed50 comp rx=0 tx=0).stop 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- 172.21.15.136:0/1026902938 shutdown_connections 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c070870 0x7fb53c0c6960 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 --2- 172.21.15.136:0/1026902938 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c143170 0x7fb53c15d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- 172.21.15.136:0/1026902938 >> 172.21.15.136:0/1026902938 conn(0x7fb53c0c9d50 msgr2=0x7fb53c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:44.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- 172.21.15.136:0/1026902938 shutdown_connections 2024-04-25T11:38:44.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- 172.21.15.136:0/1026902938 wait complete. 2024-04-25T11:38:44.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 Processor -- start 2024-04-25T11:38:44.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.710+0000 7fb54200d640 1 -- start start 2024-04-25T11:38:44.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c070870 0x7fb53c15c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb53c070c80 0x7fb53c155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c15e2e0 con 0x7fb53c143170 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c070350 con 0x7fb53c070c80 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb53c0712a0 con 0x7fb53c070870 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:32788/0 (socket says 172.21.15.136:32788) 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 -- 172.21.15.136:0/3762480951 learned_addr learned my addr 172.21.15.136:0/3762480951 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53b7fe640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c070870 0x7fb53c15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53affd640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb53c070c80 0x7fb53c155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 -- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c070870 msgr2=0x7fb53c15c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c070870 0x7fb53c15c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 -- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb53c070c80 msgr2=0x7fb53c155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb53c070c80 0x7fb53c155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 -- 172.21.15.136:0/3762480951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c15c890 con 0x7fb53c143170 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb53bfff640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c730 tx=0x7fb53400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534003b80 con 0x7fb53c143170 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb534003d20 con 0x7fb53c143170 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c15b2b0 con 0x7fb53c143170 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53400aa30 con 0x7fb53c143170 2024-04-25T11:38:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c15b870 con 0x7fb53c143170 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c15ca60 con 0x7fb53c143170 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb53400abd0 con 0x7fb53c143170 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 0x7fb528071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.714+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fb53408f7d0 con 0x7fb53c143170 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.718+0000 7fb53b7fe640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 0x7fb528071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:44.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.718+0000 7fb53b7fe640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 0x7fb528071910 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb530009a50 tx=0x7fb530008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:44.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.722+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb53c143580 con 0x7fb53c143170 2024-04-25T11:38:44.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.910+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb53c06a350 con 0x7fb53c143170 2024-04-25T11:38:44.896 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.910+0000 7fb523fff640 1 -- 172.21.15.136:0/3762480951 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+6104 (secure 0 0 0) 0x7fb534060e20 con 0x7fb53c143170 2024-04-25T11:38:44.897 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:44.897 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5385,5408,5449,5517,6246]},"id":1}]} 2024-04-25T11:38:44.897 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 78 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 msgr2=0x7fb528071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 0x7fb528071910 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb530009a50 tx=0x7fb530008040 comp rx=0 tx=0).stop 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 msgr2=0x7fb53c138ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb53400c730 tx=0x7fb53400cbf0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 shutdown_connections 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb53c143170 0x7fb53c138ae0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb53c070c80 0x7fb53c155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb52806f400 0x7fb528071910 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 --2- 172.21.15.136:0/3762480951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb53c070870 0x7fb53c15c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:44.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 >> 172.21.15.136:0/3762480951 conn(0x7fb53c0c9d50 msgr2=0x7fb53c1424c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:44.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 shutdown_connections 2024-04-25T11:38:44.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:44.914+0000 7fb54200d640 1 -- 172.21.15.136:0/3762480951 wait complete. 2024-04-25T11:38:44.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T11:38:46.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:38:47.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 Processor -- start 2024-04-25T11:38:47.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 -- start start 2024-04-25T11:38:47.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f16380b5ba0 0x7f16380ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16380ccb80 con 0x7f1638156330 2024-04-25T11:38:47.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1638156740 con 0x7f16380cdee0 2024-04-25T11:38:47.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163ed84640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f16381602f0 con 0x7f16380b5ba0 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f16380b5ba0 0x7f16380ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f16380b5ba0 0x7f16380ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:32790/0 (socket says 172.21.15.136:32790) 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 -- 172.21.15.136:0/36575217 learned_addr learned my addr 172.21.15.136:0/36575217 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163cb20640 1 --1- 172.21.15.136:0/36575217 >> v1:172.21.15.136:6789/0 conn(0x7f1638156330 0x7f163815cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:45884/0 (socket says 172.21.15.136:45884) 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 -- 172.21.15.136:0/36575217 >> v1:172.21.15.157:6789/0 conn(0x7f16380cdee0 legacy=0x7f16380b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 -- 172.21.15.136:0/36575217 >> v1:172.21.15.136:6789/0 conn(0x7f1638156330 legacy=0x7f163815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f163d321640 1 -- 172.21.15.136:0/36575217 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163815f6e0 con 0x7f16380b5ba0 2024-04-25T11:38:47.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.098+0000 7f16377fe640 1 -- 172.21.15.136:0/36575217 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786686863 0 0) 0x7f16380ccb80 con 0x7f1638156330 2024-04-25T11:38:47.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163d321640 1 --2- 172.21.15.136:0/36575217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380ce2f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1630009e50 tx=0x7f163001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258a516221a15622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f16377fe640 1 -- 172.21.15.136:0/36575217 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300b4070 con 0x7f16380b5ba0 2024-04-25T11:38:47.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f16377fe640 1 -- 172.21.15.136:0/36575217 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16300af400 con 0x7f16380b5ba0 2024-04-25T11:38:47.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f16377fe640 1 -- 172.21.15.136:0/36575217 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300af7d0 con 0x7f16380b5ba0 2024-04-25T11:38:47.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 -- 172.21.15.136:0/36575217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 msgr2=0x7f16380ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 --2- 172.21.15.136:0/36575217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380ce2f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1630009e50 tx=0x7f163001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 -- 172.21.15.136:0/36575217 shutdown_connections 2024-04-25T11:38:47.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 --2- 172.21.15.136:0/36575217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380ce2f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 -- 172.21.15.136:0/36575217 >> 172.21.15.136:0/36575217 conn(0x7f163806d6c0 msgr2=0x7f163806e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:47.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 -- 172.21.15.136:0/36575217 shutdown_connections 2024-04-25T11:38:47.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.102+0000 7f163ed84640 1 -- 172.21.15.136:0/36575217 wait complete. 2024-04-25T11:38:47.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 Processor -- start 2024-04-25T11:38:47.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- start start 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16380cdee0 0x7f1638154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1638156330 0x7f163814db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f163815ff80 con 0x7f16380b5ba0 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f163815fca0 con 0x7f1638156330 2024-04-25T11:38:47.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f16380cd320 con 0x7f16380cdee0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:32804/0 (socket says 172.21.15.136:32804) 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 -- 172.21.15.136:0/2146800612 learned_addr learned my addr 172.21.15.136:0/2146800612 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 -- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16380cdee0 msgr2=0x7f1638154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163d321640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1638156330 0x7f163814db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16380cdee0 0x7f1638154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 -- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1638156330 msgr2=0x7f163814db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1638156330 0x7f163814db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 -- 172.21.15.136:0/2146800612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16380d19f0 con 0x7f16380b5ba0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163cb20640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f162800c440 tx=0x7f162800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628018070 con 0x7f16380b5ba0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16280074e0 con 0x7f16380b5ba0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16380cb7d0 con 0x7f16380b5ba0 2024-04-25T11:38:47.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628004a10 con 0x7f16380b5ba0 2024-04-25T11:38:47.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16380cf2f0 con 0x7f16380b5ba0 2024-04-25T11:38:47.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.106+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163806a350 con 0x7f16380b5ba0 2024-04-25T11:38:47.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.110+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f162801b070 con 0x7f16380b5ba0 2024-04-25T11:38:47.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.110+0000 7f16357fa640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 0x7f1624071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:38:47.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.110+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f162808ebd0 con 0x7f16380b5ba0 2024-04-25T11:38:47.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.110+0000 7f1637fff640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 0x7f1624071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:38:47.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.110+0000 7f1637fff640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 0x7f1624071910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16380ce980 tx=0x7f162c0079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:38:47.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.114+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f162805b480 con 0x7f16380b5ba0 2024-04-25T11:38:47.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16380cd9c0 con 0x7f16380b5ba0 2024-04-25T11:38:47.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f16357fa640 1 -- 172.21.15.136:0/2146800612 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7f162805ad80 con 0x7f16380b5ba0 2024-04-25T11:38:47.304 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:38:47.304 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:38:47.304 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:38:47.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 msgr2=0x7f1624071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 0x7f1624071910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16380ce980 tx=0x7f162c0079c0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 msgr2=0x7f16380d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.318+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f162800c440 tx=0x7f162800c900 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 shutdown_connections 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1638156330 0x7f163814db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f162406f400 0x7f1624071910 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f16380cdee0 0x7f1638154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 --2- 172.21.15.136:0/2146800612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16380b5ba0 0x7f16380d14b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:38:47.307 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 >> 172.21.15.136:0/2146800612 conn(0x7f163806d6c0 msgr2=0x7f163815c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:38:47.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 shutdown_connections 2024-04-25T11:38:47.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:38:47.322+0000 7f163ed84640 1 -- 172.21.15.136:0/2146800612 wait complete. 2024-04-25T11:38:47.331 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-04-25T11:38:47.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2024-04-25T11:38:49.303 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-04-25T11:38:49.316+0000 7f9941f69640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T11:38:49.303 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-04-25T11:38:49.316+0000 7f9941f69640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:38:49.303 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-04-25T11:38:49.316+0000 7f9941f69640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:38:49.304 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-04-25T11:38:49.316+0000 7f9941f69640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:38:49.304 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-04-25T11:38:49.316+0000 7f9941f69640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:38:54.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:38:54.462 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:39:01.518 INFO:tasks.workunit.client.0.smithi136.stdout:fs setup took 948 secs 2024-04-25T11:39:02.133 INFO:tasks.workunit.client.0.smithi136.stdout:Syncing()...0 sec 2024-04-25T11:39:02.134 INFO:tasks.workunit.client.0.smithi136.stdout:Starting Actual Benchmark At: Thu Apr 25 11:39:02 2024 2024-04-25T11:39:02.134 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:39:07.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:07.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 Processor -- start 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 -- start start 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9e000c82e0 0x7f9e00160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e00072da0 0x7f9e000731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e001648a0 con 0x7f9e000ca160 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9e0015ab80 con 0x7f9e00072da0 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e06841640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9e00161350 con 0x7f9e000c82e0 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e0503e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e00072da0 0x7f9e000731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e0503e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e00072da0 0x7f9e000731b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38418/0 (socket says 172.21.15.136:38418) 2024-04-25T11:39:07.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.957+0000 7f9e0503e640 1 -- 172.21.15.136:0/1217374732 learned_addr learned my addr 172.21.15.136:0/1217374732 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:07.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.961+0000 7f9e0503e640 1 -- 172.21.15.136:0/1217374732 >> v2:172.21.15.157:3300/0 conn(0x7f9e000c82e0 msgr2=0x7f9e00160e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:07.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.961+0000 7f9e0503e640 1 --2- 172.21.15.136:0/1217374732 >> v2:172.21.15.157:3300/0 conn(0x7f9e000c82e0 0x7f9e00160e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.961+0000 7f9e0503e640 1 -- 172.21.15.136:0/1217374732 >> v1:172.21.15.136:6789/0 conn(0x7f9e000ca160 legacy=0x7f9e000701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.961+0000 7f9e0503e640 1 -- 172.21.15.136:0/1217374732 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000c8a20 con 0x7f9e00072da0 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.965+0000 7f9e0503e640 1 --2- 172.21.15.136:0/1217374732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000731b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9e00071290 tx=0x7f9df001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb140c91563e2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.965+0000 7f9e0483d640 1 -- 172.21.15.136:0/1217374732 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df00b1070 con 0x7f9e00072da0 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.965+0000 7f9e0483d640 1 -- 172.21.15.136:0/1217374732 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df00ac410 con 0x7f9e00072da0 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.965+0000 7f9e06841640 1 -- 172.21.15.136:0/1217374732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 msgr2=0x7f9e000731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:07.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.965+0000 7f9e06841640 1 --2- 172.21.15.136:0/1217374732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000731b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9e00071290 tx=0x7f9df001c3d0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 -- 172.21.15.136:0/1217374732 shutdown_connections 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 --2- 172.21.15.136:0/1217374732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000731b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 --2- 172.21.15.136:0/1217374732 >> v2:172.21.15.157:3300/0 conn(0x7f9e000c82e0 0x7f9e00160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 -- 172.21.15.136:0/1217374732 >> 172.21.15.136:0/1217374732 conn(0x7f9e0006c840 msgr2=0x7f9e0006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 -- 172.21.15.136:0/1217374732 shutdown_connections 2024-04-25T11:39:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.969+0000 7f9e06841640 1 -- 172.21.15.136:0/1217374732 wait complete. 2024-04-25T11:39:07.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 Processor -- start 2024-04-25T11:39:07.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 -- start start 2024-04-25T11:39:07.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e000c82e0 0x7f9e00162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e000ca160 0x7f9e00162ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001640a0 con 0x7f9e00072da0 2024-04-25T11:39:07.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9e00163dc0 con 0x7f9e000ca160 2024-04-25T11:39:07.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06841640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9e00164380 con 0x7f9e000c82e0 2024-04-25T11:39:07.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:07.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0503e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e000c82e0 0x7f9e00162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:07.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38434/0 (socket says 172.21.15.136:38434) 2024-04-25T11:39:07.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 -- 172.21.15.136:0/287009513 learned_addr learned my addr 172.21.15.136:0/287009513 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e06040640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e000ca160 0x7f9e00162ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 -- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e000c82e0 msgr2=0x7f9e00162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e000c82e0 0x7f9e00162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 -- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e000ca160 msgr2=0x7f9e00162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e000ca160 0x7f9e00162ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 -- 172.21.15.136:0/287009513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000c68e0 con 0x7f9e00072da0 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.973+0000 7f9e0583f640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00b420 tx=0x7f9dfc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:07.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.977+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc011830 con 0x7f9e00072da0 2024-04-25T11:39:07.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.977+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc016070 con 0x7f9e00072da0 2024-04-25T11:39:07.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e001615b0 con 0x7f9e00072da0 2024-04-25T11:39:07.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e000747b0 con 0x7f9e00072da0 2024-04-25T11:39:07.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.977+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc011c30 con 0x7f9e00072da0 2024-04-25T11:39:07.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.981+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9dfc01b460 con 0x7f9e00072da0 2024-04-25T11:39:07.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.981+0000 7f9df67fc640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 0x7f9de40719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:07.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.981+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f9dfc08f650 con 0x7f9e00072da0 2024-04-25T11:39:07.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.981+0000 7f9e0503e640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 0x7f9de40719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:07.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:07.981+0000 7f9e0503e640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 0x7f9de40719e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9df00ae000 tx=0x7f9df00a6040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:07.990 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.009+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de80052a0 con 0x7f9e00072da0 2024-04-25T11:39:08.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.017+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9dfc05b770 con 0x7f9e00072da0 2024-04-25T11:39:08.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.961+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9de8005590 con 0x7f9e00072da0 2024-04-25T11:39:08.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.961+0000 7f9df67fc640 1 -- 172.21.15.136:0/287009513 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7f9dfc05fa70 con 0x7f9e00072da0 2024-04-25T11:39:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:08.949 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:39:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:39:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 msgr2=0x7f9de40719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 0x7f9de40719e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9df00ae000 tx=0x7f9df00a6040 comp rx=0 tx=0).stop 2024-04-25T11:39:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 msgr2=0x7f9e000c63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00b420 tx=0x7f9dfc00b8e0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 shutdown_connections 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e000ca160 0x7f9e00162ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9de406f4d0 0x7f9de40719e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9e000c82e0 0x7f9e00162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 --2- 172.21.15.136:0/287009513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e00072da0 0x7f9e000c63a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 >> 172.21.15.136:0/287009513 conn(0x7f9e0006c840 msgr2=0x7f9e00158820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 shutdown_connections 2024-04-25T11:39:08.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:08.977+0000 7f9e06841640 1 -- 172.21.15.136:0/287009513 wait complete. 2024-04-25T11:39:09.052 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=3) 2024-04-25T11:39:09.052 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T11:39:09.052 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:39:09.078 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T11:39:09.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:09.588 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.605+0000 7fdd9e8e3640 1 Processor -- start 2024-04-25T11:39:09.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.605+0000 7fdd9e8e3640 1 -- start start 2024-04-25T11:39:09.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e8e3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdd980743b0 0x7fdd980701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e8e3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981628d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e8e3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd98158300 con 0x7fdd980c7940 2024-04-25T11:39:09.593 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e8e3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fdd980c54f0 con 0x7fdd980743b0 2024-04-25T11:39:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e8e3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd98073f20 con 0x7fdd98070700 2024-04-25T11:39:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9d8e1640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fdd980c7940 0x7fdd98152050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:47640/0 (socket says 172.21.15.136:47640) 2024-04-25T11:39:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9d8e1640 1 -- 172.21.15.136:0/3449432466 learned_addr learned my addr 172.21.15.136:0/3449432466 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:39:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.609+0000 7fdd9e0e2640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.613+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/3449432466 >> v2:172.21.15.136:3300/0 conn(0x7fdd980743b0 0x7fdd980701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.613+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/3449432466 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 msgr2=0x7fdd981628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:09.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.613+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981628d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.613+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/3449432466 >> v1:172.21.15.157:6789/0 conn(0x7fdd980c7940 legacy=0x7fdd98152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.613+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/3449432466 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd98158060 con 0x7fdd980743b0 2024-04-25T11:39:09.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.621+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd980701c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009d90 tx=0x7fdd9001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7967c597b21daab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.621+0000 7fdd9c8df640 1 -- 172.21.15.136:0/3449432466 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd900b2070 con 0x7fdd980743b0 2024-04-25T11:39:09.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.621+0000 7fdd9c8df640 1 -- 172.21.15.136:0/3449432466 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd900a5870 con 0x7fdd980743b0 2024-04-25T11:39:09.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.621+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/3449432466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 msgr2=0x7fdd980701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.621+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd980701c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009d90 tx=0x7fdd9001c9b0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.629+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/3449432466 shutdown_connections 2024-04-25T11:39:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.629+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981628d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.629+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/3449432466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd980701c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.629+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/3449432466 >> 172.21.15.136:0/3449432466 conn(0x7fdd9806c840 msgr2=0x7fdd9806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:09.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.633+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/3449432466 shutdown_connections 2024-04-25T11:39:09.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.633+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/3449432466 wait complete. 2024-04-25T11:39:09.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.633+0000 7fdd9e8e3640 1 Processor -- start 2024-04-25T11:39:09.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.633+0000 7fdd9e8e3640 1 -- start start 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd980c7940 0x7fdd98140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980c9470 con 0x7fdd980743b0 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98073bb0 con 0x7fdd980c7940 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e8e3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdd98071210 con 0x7fdd98070700 2024-04-25T11:39:09.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38456/0 (socket says 172.21.15.136:38456) 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/1181716130 learned_addr learned my addr 172.21.15.136:0/1181716130 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9e0e2640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd980c7940 0x7fdd98140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d8e1640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 msgr2=0x7fdd981464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd980c7940 msgr2=0x7fdd98140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd980c7940 0x7fdd98140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 -- 172.21.15.136:0/1181716130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd98157750 con 0x7fdd980743b0 2024-04-25T11:39:09.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.637+0000 7fdd9d0e0640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdd9001a5e0 tx=0x7fdd90019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.641+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd900b2040 con 0x7fdd980743b0 2024-04-25T11:39:09.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.641+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd981469f0 con 0x7fdd980743b0 2024-04-25T11:39:09.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.641+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd98156d30 con 0x7fdd980743b0 2024-04-25T11:39:09.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.641+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9806a2e0 con 0x7fdd980743b0 2024-04-25T11:39:09.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd900bf070 con 0x7fdd980743b0 2024-04-25T11:39:09.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd900b8470 con 0x7fdd980743b0 2024-04-25T11:39:09.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd900a5a60 con 0x7fdd980743b0 2024-04-25T11:39:09.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd8e7fc640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 0x7fdd7c071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:09.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fdd9012b1c0 con 0x7fdd980743b0 2024-04-25T11:39:09.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.649+0000 7fdd9d8e1640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 0x7fdd7c071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:09.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.657+0000 7fdd9d8e1640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 0x7fdd7c071a40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdd88004910 tx=0x7fdd8800a210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:09.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:09.661+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdd900165a0 con 0x7fdd980743b0 2024-04-25T11:39:10.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.489+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd98155ba0 con 0x7fdd980743b0 2024-04-25T11:39:10.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.489+0000 7fdd8e7fc640 1 -- 172.21.15.136:0/1181716130 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7fdd900f7ae0 con 0x7fdd980743b0 2024-04-25T11:39:10.472 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:10.472 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:39:10.472 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 msgr2=0x7fdd7c071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 0x7fdd7c071a40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdd88004910 tx=0x7fdd8800a210 comp rx=0 tx=0).stop 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 msgr2=0x7fdd98140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fdd9001a5e0 tx=0x7fdd90019750 comp rx=0 tx=0).stop 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 shutdown_connections 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdd980c7940 0x7fdd98140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd980743b0 0x7fdd98140a70 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fdd7c06f530 0x7fdd7c071a40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 --2- 172.21.15.136:0/1181716130 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdd98070700 0x7fdd981464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 >> 172.21.15.136:0/1181716130 conn(0x7fdd9806c840 msgr2=0x7fdd980c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 shutdown_connections 2024-04-25T11:39:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:10.513+0000 7fdd9e8e3640 1 -- 172.21.15.136:0/1181716130 wait complete. 2024-04-25T11:39:10.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:39:12.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:12.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.961+0000 7fd8b0929640 1 Processor -- start 2024-04-25T11:39:12.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.961+0000 7fd8b0929640 1 -- start start 2024-04-25T11:39:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8b0929640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ac0c2bb0 0x7fd8ac0c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8b0929640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0c7b30 con 0x7fd8ac073300 2024-04-25T11:39:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8b0929640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac0c9630 con 0x7fd8ac16cf90 2024-04-25T11:39:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8b0929640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ac0c93e0 con 0x7fd8ac0c2bb0 2024-04-25T11:39:12.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a1d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd8ac16cf90 0x7fd8ac0c20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:47644/0 (socket says 172.21.15.136:47644) 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a1d9b640 1 -- 172.21.15.136:0/3661508043 learned_addr learned my addr 172.21.15.136:0/3661508043 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1800471555 0 0) 0x7fd8ac0c7b30 con 0x7fd8ac073300 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd898003530 con 0x7fd8ac073300 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374927396 0 0) 0x7fd8ac0c9630 con 0x7fd8ac16cf90 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8ac0c7b30 con 0x7fd8ac16cf90 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3262377298 0 0) 0x7fd898003530 con 0x7fd8ac073300 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ac0c2bb0 msgr2=0x7fd8ac0c85d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ac0c2bb0 0x7fd8ac0c85d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 >> v1:172.21.15.157:6789/0 conn(0x7fd8ac16cf90 legacy=0x7fd8ac0c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac0c7dd0 con 0x7fd8ac073300 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd8a4001980 con 0x7fd8ac073300 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 >> v1:172.21.15.136:6789/0 conn(0x7fd8ac073300 legacy=0x7fd8ac1736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd898003b80 0x7fd898003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd898004810 0x7fd898004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd89800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac0c93e0 con 0x7fd898003b80 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd89800c1c0 con 0x7fd898004810 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd89800c3c0 con 0x7fd898005360 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd89800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 -- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd898004810 msgr2=0x7fd898004c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd898004810 0x7fd898004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 -- 172.21.15.136:0/3661508043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd898003b80 msgr2=0x7fd898003fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd898003b80 0x7fd898003fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.965+0000 7fd8aad9d640 1 -- 172.21.15.136:0/3661508043 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac0c7dd0 con 0x7fd898005360 2024-04-25T11:39:12.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd89800b9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0739b0 tx=0x7fd8a401cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39bb0121e1a37858 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:12.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40b2070 con 0x7fd898005360 2024-04-25T11:39:12.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a40a5c70 con 0x7fd898005360 2024-04-25T11:39:12.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8a9d9b640 1 -- 172.21.15.136:0/3661508043 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a400c0e0 con 0x7fd898005360 2024-04-25T11:39:12.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8b0929640 1 -- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 msgr2=0x7fd89800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8b0929640 1 --2- 172.21.15.136:0/3661508043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd89800b9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0739b0 tx=0x7fd8a401cec0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.973+0000 7fd8aad9d640 1 -- 172.21.15.136:0/3661508043 reap_dead start 2024-04-25T11:39:12.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.981+0000 7fd8b0929640 1 -- 172.21.15.136:0/3661508043 shutdown_connections 2024-04-25T11:39:12.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.981+0000 7fd8b0929640 1 -- 172.21.15.136:0/3661508043 >> 172.21.15.136:0/3661508043 conn(0x7fd8ac0cdae0 msgr2=0x7fd8ac0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:12.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.981+0000 7fd8b0929640 1 -- 172.21.15.136:0/3661508043 shutdown_connections 2024-04-25T11:39:12.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.981+0000 7fd8b0929640 1 -- 172.21.15.136:0/3661508043 wait complete. 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 Processor -- start 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 -- start start 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd8ac156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ac0b2760 0x7fd8ac1621c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8a1d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ac0b2760 0x7fd8ac1621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38468/0 (socket says 172.21.15.136:38468) 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 -- 172.21.15.136:0/2184181231 learned_addr learned my addr 172.21.15.136:0/2184181231 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:12.970 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac074040 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac073ec0 con 0x7fd8ac0b2760 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ac0c85d0 con 0x7fd898005360 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 -- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 msgr2=0x7fd8ac156a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd8ac156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 -- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ac0b2760 msgr2=0x7fd8ac1621c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ac0b2760 0x7fd8ac1621c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.985+0000 7fd8aad9d640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac0b1380 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:12.989+0000 7fd8aad9d640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd8a401c930 tx=0x7fd898005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:12.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40b2040 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a40bb070 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ac0b1620 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ac0b1c80 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4001900 con 0x7fd8ac15e6d0 2024-04-25T11:39:12.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.013+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ac06a2e0 con 0x7fd8ac15e6d0 2024-04-25T11:39:13.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.017+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd8a40b5070 con 0x7fd8ac15e6d0 2024-04-25T11:39:13.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.017+0000 7fd8a37fe640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 0x7fd88c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:13.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.021+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7fd8a4129790 con 0x7fd8ac15e6d0 2024-04-25T11:39:13.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.025+0000 7fd8aa59c640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 0x7fd88c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:13.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.029+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd8a40a5e20 con 0x7fd8ac15e6d0 2024-04-25T11:39:13.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:13.029+0000 7fd8aa59c640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 0x7fd88c071a10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac06f7c0 tx=0x7fd89c006af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:14.058 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.073+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8ac159a00 con 0x7fd8ac15e6d0 2024-04-25T11:39:14.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.077+0000 7fd8a37fe640 1 -- 172.21.15.136:0/2184181231 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7fd8a4016bd0 con 0x7fd8ac15e6d0 2024-04-25T11:39:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:14.061 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:39:14.062 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:39:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.093+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 msgr2=0x7fd88c071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.093+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 0x7fd88c071a10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac06f7c0 tx=0x7fd89c006af0 comp rx=0 tx=0).stop 2024-04-25T11:39:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.093+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 msgr2=0x7fd8ac162ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:14.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.093+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd8a401c930 tx=0x7fd898005840 comp rx=0 tx=0).stop 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 shutdown_connections 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8ac15e6d0 0x7fd8ac162ff0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ac0b2760 0x7fd8ac1621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd88c06f500 0x7fd88c071a10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 --2- 172.21.15.136:0/2184181231 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd898005360 0x7fd8ac156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.097+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 >> 172.21.15.136:0/2184181231 conn(0x7fd8ac0cdae0 msgr2=0x7fd8ac15c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:14.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.101+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 shutdown_connections 2024-04-25T11:39:14.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:14.101+0000 7fd8b0929640 1 -- 172.21.15.136:0/2184181231 wait complete. 2024-04-25T11:39:14.193 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:39:16.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:16.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.745+0000 7f95cdc5d640 1 Processor -- start 2024-04-25T11:39:16.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 -- start start 2024-04-25T11:39:16.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95c8073300 0x7f95c80c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 0x7f95c816cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95c80c7b30 con 0x7f95c80c2bb0 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f95c80c93e0 con 0x7f95c8073300 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95cdc5d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95c80c2990 con 0x7f95c816c870 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95c77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95c8073300 0x7f95c80c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95c77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95c8073300 0x7f95c80c85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36616/0 (socket says 172.21.15.136:36616) 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.749+0000 7f95c77fe640 1 -- 172.21.15.136:0/2199556175 learned_addr learned my addr 172.21.15.136:0/2199556175 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.757+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956637707 0 0) 0x7f95c80c7b30 con 0x7f95c80c2bb0 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95b0003510 con 0x7f95c80c2bb0 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.757+0000 7f95c7fff640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 0x7f95c816cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:16.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c77fe640 1 -- 172.21.15.136:0/2199556175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 msgr2=0x7f95c816cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c77fe640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 0x7f95c816cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3926091898 0 0) 0x7f95b0003510 con 0x7f95c80c2bb0 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c77fe640 1 -- 172.21.15.136:0/2199556175 >> v1:172.21.15.157:6789/0 conn(0x7f95c80c2bb0 legacy=0x7f95c80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c77fe640 1 -- 172.21.15.136:0/2199556175 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c80c9630 con 0x7f95c8073300 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.769+0000 7f95c7fff640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 0x7f95c816cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.773+0000 7f95c77fe640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c8073300 0x7f95c80c85d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f95b80083f0 tx=0x7f95b801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf8185d0a321401 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.773+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b80b0070 con 0x7f95c8073300 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.773+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95b80089e0 con 0x7f95c8073300 2024-04-25T11:39:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.773+0000 7f95c67fc640 1 -- 172.21.15.136:0/2199556175 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0016a0 con 0x7f95c8073300 2024-04-25T11:39:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.777+0000 7f95cdc5d640 1 -- 172.21.15.136:0/2199556175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c8073300 msgr2=0x7f95c80c85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:16.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.777+0000 7f95cdc5d640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c8073300 0x7f95c80c85d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f95b80083f0 tx=0x7f95b801c3e0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 -- 172.21.15.136:0/2199556175 shutdown_connections 2024-04-25T11:39:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c816c870 0x7f95c816cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 --2- 172.21.15.136:0/2199556175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c8073300 0x7f95c80c85d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 -- 172.21.15.136:0/2199556175 >> 172.21.15.136:0/2199556175 conn(0x7f95c80cdae0 msgr2=0x7f95c80c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 -- 172.21.15.136:0/2199556175 shutdown_connections 2024-04-25T11:39:16.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.781+0000 7f95cdc5d640 1 -- 172.21.15.136:0/2199556175 wait complete. 2024-04-25T11:39:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.785+0000 7f95cdc5d640 1 Processor -- start 2024-04-25T11:39:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.785+0000 7f95cdc5d640 1 -- start start 2024-04-25T11:39:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 0x7f95c8205330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95c816c870 0x7f95c80afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f95c80c8eb0 con 0x7f95c80c2bb0 2024-04-25T11:39:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95c7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95c7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:52858/0 (socket says 172.21.15.136:52858) 2024-04-25T11:39:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95c7fff640 1 -- 172.21.15.136:0/16562756 learned_addr learned my addr 172.21.15.136:0/16562756 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:16.776 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95c816d1c0 con 0x7f95c816c870 2024-04-25T11:39:16.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95c80c8b10 con 0x7f95c80b25b0 2024-04-25T11:39:16.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.793+0000 7f95c77fe640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 0x7f95c8205330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:16.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c7fff640 1 -- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95c816c870 msgr2=0x7f95c80afc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:39:16.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c7fff640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95c816c870 0x7f95c80afc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c7fff640 1 -- 172.21.15.136:0/16562756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 msgr2=0x7f95c8205330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:16.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c7fff640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 0x7f95c8205330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:16.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c7fff640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c8205870 con 0x7f95c80b25b0 2024-04-25T11:39:16.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.797+0000 7f95c77fe640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 0x7f95c8205330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:16.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.801+0000 7f95c7fff640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95c000cd80 tx=0x7f95c000a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:16.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0003ba0 con 0x7f95c80b25b0 2024-04-25T11:39:16.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c0003d60 con 0x7f95c80b25b0 2024-04-25T11:39:16.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c80c4840 con 0x7f95c80b25b0 2024-04-25T11:39:16.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0004260 con 0x7f95c80b25b0 2024-04-25T11:39:16.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c80b1d30 con 0x7f95c80b25b0 2024-04-25T11:39:16.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f95c0004400 con 0x7f95c80b25b0 2024-04-25T11:39:16.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 0x7f95b0082c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:16.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f95c0090260 con 0x7f95c80b25b0 2024-04-25T11:39:16.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.809+0000 7f95c77fe640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 0x7f95b0082c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:16.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.813+0000 7f95c77fe640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 0x7f95b0082c70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f95b80ad000 tx=0x7f95b8004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:16.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.821+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c806a2e0 con 0x7f95c80b25b0 2024-04-25T11:39:16.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:16.837+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f95c0014030 con 0x7f95c80b25b0 2024-04-25T11:39:17.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.577+0000 7f95cdc5d640 1 -- 172.21.15.136:0/16562756 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95c80af9f0 con 0x7f95c80b25b0 2024-04-25T11:39:17.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.581+0000 7f95affff640 1 -- 172.21.15.136:0/16562756 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7f95c005c510 con 0x7f95c80b25b0 2024-04-25T11:39:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:17.568 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:39:17.568 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.609+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 msgr2=0x7f95b0082c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.609+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 0x7f95b0082c70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f95b80ad000 tx=0x7f95b8004740 comp rx=0 tx=0).stop 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 msgr2=0x7f95c80cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95c000cd80 tx=0x7f95c000a650 comp rx=0 tx=0).stop 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 shutdown_connections 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f95b0080760 0x7f95b0082c70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:17.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95c80b25b0 0x7f95c80cc2e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:17.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.613+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95c816c870 0x7f95c80afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:17.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.617+0000 7f95adffb640 1 --2- 172.21.15.136:0/16562756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95c80c2bb0 0x7f95c8205330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:17.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.621+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 >> 172.21.15.136:0/16562756 conn(0x7f95c80cdae0 msgr2=0x7f95c8074690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:17.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.621+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 shutdown_connections 2024-04-25T11:39:17.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:17.621+0000 7f95adffb640 1 -- 172.21.15.136:0/16562756 wait complete. 2024-04-25T11:39:17.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:39:19.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 Processor -- start 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 -- start start 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec08d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ec0c9740 con 0x7f78ec168c50 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78ec0ca4b0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78f12fe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec0c8ba0 con 0x7f78ec0c7c40 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78eb7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f78ec168c50 0x7f78ec0c9030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:43998/0 (socket says 172.21.15.136:43998) 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.329+0000 7f78eb7fe640 1 -- 172.21.15.136:0/1514471588 learned_addr learned my addr 172.21.15.136:0/1514471588 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.333+0000 7f78ebfff640 1 --2- 172.21.15.136:0/1514471588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec08d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:20.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.349+0000 7f78ebfff640 1 -- 172.21.15.136:0/1514471588 >> v1:172.21.15.157:6789/0 conn(0x7f78ec0c99b0 legacy=0x7f78ec1734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:20.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.349+0000 7f78ebfff640 1 -- 172.21.15.136:0/1514471588 >> v1:172.21.15.136:6789/0 conn(0x7f78ec168c50 legacy=0x7f78ec0c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:20.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.349+0000 7f78ebfff640 1 -- 172.21.15.136:0/1514471588 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec169200 con 0x7f78ec0c7c40 2024-04-25T11:39:20.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.333+0000 7f78eaffd640 1 -- 172.21.15.136:0/1514471588 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138562013 0 0) 0x7f78ec0c9740 con 0x7f78ec168c50 2024-04-25T11:39:20.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.353+0000 7f78ebfff640 1 --2- 172.21.15.136:0/1514471588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec08d8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78e40097a0 tx=0x7f78e4020c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c63b4b39a4c71ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:20.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.353+0000 7f78eaffd640 1 -- 172.21.15.136:0/1514471588 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40a9e40 con 0x7f78ec0c7c40 2024-04-25T11:39:20.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.361+0000 7f78eaffd640 1 -- 172.21.15.136:0/1514471588 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e40090d0 con 0x7f78ec0c7c40 2024-04-25T11:39:20.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.361+0000 7f78f12fe640 1 -- 172.21.15.136:0/1514471588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 msgr2=0x7f78ec08d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:20.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.361+0000 7f78f12fe640 1 --2- 172.21.15.136:0/1514471588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec08d8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78e40097a0 tx=0x7f78e4020c90 comp rx=0 tx=0).stop 2024-04-25T11:39:20.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.369+0000 7f78f12fe640 1 -- 172.21.15.136:0/1514471588 shutdown_connections 2024-04-25T11:39:20.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.369+0000 7f78f12fe640 1 --2- 172.21.15.136:0/1514471588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec08d8b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:20.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.369+0000 7f78f12fe640 1 -- 172.21.15.136:0/1514471588 >> 172.21.15.136:0/1514471588 conn(0x7f78ec06d100 msgr2=0x7f78ec070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:20.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.373+0000 7f78f12fe640 1 -- 172.21.15.136:0/1514471588 shutdown_connections 2024-04-25T11:39:20.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.373+0000 7f78f12fe640 1 -- 172.21.15.136:0/1514471588 wait complete. 2024-04-25T11:39:20.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.373+0000 7f78f12fe640 1 Processor -- start 2024-04-25T11:39:20.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.373+0000 7f78f12fe640 1 -- start start 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec167150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78ec168c50 0x7f78ec1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec169aa0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36630/0 (socket says 172.21.15.136:36630) 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 -- 172.21.15.136:0/3495006245 learned_addr learned my addr 172.21.15.136:0/3495006245 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78ebfff640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec167150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f78ec1697c0 con 0x7f78ec168c50 2024-04-25T11:39:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f12fe640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78ec0c8830 con 0x7f78ec0c7c40 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78f0afd640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78ec168c50 0x7f78ec1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 -- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 msgr2=0x7f78ec167150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec167150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 -- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78ec168c50 msgr2=0x7f78ec1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78ec168c50 0x7f78ec1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ec1f40d0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78eb7fe640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c410 tx=0x7f78e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:20.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.377+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00074e0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.381+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e00047f0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.381+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0007680 con 0x7f78ec0c99b0 2024-04-25T11:39:20.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.381+0000 7f78f12fe640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ec1fff10 con 0x7f78ec0c99b0 2024-04-25T11:39:20.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.381+0000 7f78f12fe640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ec1f3390 con 0x7f78ec0c99b0 2024-04-25T11:39:20.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.381+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f78e0011070 con 0x7f78ec0c99b0 2024-04-25T11:39:20.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.385+0000 7f78e8ff9640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 0x7f78d8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:20.367 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.385+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7595+0+0 (secure 0 0 0) 0x7f78e008f470 con 0x7f78ec0c99b0 2024-04-25T11:39:20.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.389+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c40052a0 con 0x7f78ec0c99b0 2024-04-25T11:39:20.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.389+0000 7f78ebfff640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 0x7f78d8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:20.373 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.389+0000 7f78ebfff640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 0x7f78d8071940 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f78ec06efe0 tx=0x7f78e40ab070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:20.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:20.409+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f78e005bc90 con 0x7f78ec0c99b0 2024-04-25T11:39:21.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.265+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78c4005860 con 0x7f78ec0c99b0 2024-04-25T11:39:21.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.265+0000 7f78e8ff9640 1 -- 172.21.15.136:0/3495006245 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+6931 (secure 0 0 0) 0x7f78e005b590 con 0x7f78ec0c99b0 2024-04-25T11:39:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:21.252 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5839,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:38:42.167286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5385,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5385":{"gid":5385,"name":"a","rank":3,"incarnation":32,"state":"up:active","state_seq":40,"addr":"172.21.15.136:6833/906354453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":906354453},{"type":"v1","addr":"172.21.15.136:6833","nonce":906354453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408,5385]},"id":1}]} 2024-04-25T11:39:21.252 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 79 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 msgr2=0x7f78d8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 0x7f78d8071940 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f78ec06efe0 tx=0x7f78e40ab070 comp rx=0 tx=0).stop 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 msgr2=0x7f78ec0cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c410 tx=0x7f78e000c8d0 comp rx=0 tx=0).stop 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 shutdown_connections 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78ec168c50 0x7f78ec1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78ec0c99b0 0x7f78ec0cb020 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f78d806f430 0x7f78d8071940 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 --2- 172.21.15.136:0/3495006245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78ec0c7c40 0x7f78ec167150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.281+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 >> 172.21.15.136:0/3495006245 conn(0x7f78ec06d100 msgr2=0x7f78ec1626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.285+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 shutdown_connections 2024-04-25T11:39:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:21.285+0000 7f78d27fc640 1 -- 172.21.15.136:0/3495006245 wait complete. 2024-04-25T11:39:21.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T11:39:23.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:24.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.033+0000 7f2af4440640 1 Processor -- start 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.033+0000 7f2af4440640 1 -- start start 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af4440640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2aec0c2e00 0x7f2aec15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af4440640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2aec15c300 0x7f2aec0749a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af4440640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aec0c3210 con 0x7f2aec074ee0 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af4440640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2aec15cae0 con 0x7f2aec15c300 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af4440640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2aec0c9d20 con 0x7f2aec0c2e00 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af29dd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2aec074ee0 0x7f2aec0c4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44004/0 (socket says 172.21.15.136:44004) 2024-04-25T11:39:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af29dd640 1 -- 172.21.15.136:0/2047720511 learned_addr learned my addr 172.21.15.136:0/2047720511 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af19db640 1 --2- 172.21.15.136:0/2047720511 >> v2:172.21.15.136:3300/0 conn(0x7f2aec15c300 0x7f2aec0749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154820842 0 0) 0x7f2aec0c3210 con 0x7f2aec074ee0 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad8003450 con 0x7f2aec074ee0 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 251805418 0 0) 0x7f2ad8003450 con 0x7f2aec074ee0 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 >> v2:172.21.15.157:3300/0 conn(0x7f2aec0c2e00 msgr2=0x7f2aec15bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 --2- 172.21.15.136:0/2047720511 >> v2:172.21.15.157:3300/0 conn(0x7f2aec0c2e00 0x7f2aec15bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 >> v2:172.21.15.136:3300/0 conn(0x7f2aec15c300 msgr2=0x7f2aec0749a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 --2- 172.21.15.136:0/2047720511 >> v2:172.21.15.136:3300/0 conn(0x7f2aec15c300 0x7f2aec0749a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec15c800 con 0x7f2aec074ee0 2024-04-25T11:39:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f2ae4001a00 con 0x7f2aec074ee0 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 >> v1:172.21.15.136:6789/0 conn(0x7f2aec074ee0 legacy=0x7f2aec0c4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ad8003ab0 0x7f2ad8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8008900 0x7f2ad800ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2ad800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec15cae0 con 0x7f2ad8008900 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0c9d20 con 0x7f2ad800b680 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad800e4f0 con 0x7f2ad8003ab0 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.037+0000 7f2af19db640 1 --2- 172.21.15.136:0/2047720511 >> v2:172.21.15.136:3300/0 conn(0x7f2aec15c300 0x7f2aec0749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2ad800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af19db640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8008900 0x7f2ad800ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 -- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ad8003ab0 msgr2=0x7f2ad8003f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:24.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ad8003ab0 0x7f2ad8003f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 -- 172.21.15.136:0/2047720511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8008900 msgr2=0x7f2ad800ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:24.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8008900 0x7f2ad800ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.041+0000 7f2af29dd640 1 -- 172.21.15.136:0/2047720511 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec15c800 con 0x7f2ad800b680 2024-04-25T11:39:24.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.045+0000 7f2af29dd640 1 -- 172.21.15.136:0/2047720511 reap_dead start 2024-04-25T11:39:24.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.045+0000 7f2af19db640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad8008900 0x7f2ad800ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:24.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.045+0000 7f2af29dd640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2ad800db90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400aa20 tx=0x7f2ae401d4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9168bf7022400712 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:24.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.045+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae400a620 con 0x7f2ad800b680 2024-04-25T11:39:24.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.045+0000 7f2af11da640 1 -- 172.21.15.136:0/2047720511 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae40ac620 con 0x7f2ad800b680 2024-04-25T11:39:24.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.049+0000 7f2af4440640 1 -- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 msgr2=0x7f2ad800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:24.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.049+0000 7f2af4440640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2ad800db90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400aa20 tx=0x7f2ae401d4e0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.049+0000 7f2af4440640 1 -- 172.21.15.136:0/2047720511 shutdown_connections 2024-04-25T11:39:24.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.049+0000 7f2af4440640 1 --2- 172.21.15.136:0/2047720511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2ad800db90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.049+0000 7f2af4440640 1 -- 172.21.15.136:0/2047720511 >> 172.21.15.136:0/2047720511 conn(0x7f2aec0ca5a0 msgr2=0x7f2aec0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:24.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.053+0000 7f2af4440640 1 -- 172.21.15.136:0/2047720511 shutdown_connections 2024-04-25T11:39:24.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.053+0000 7f2af4440640 1 -- 172.21.15.136:0/2047720511 wait complete. 2024-04-25T11:39:24.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 Processor -- start 2024-04-25T11:39:24.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 -- start start 2024-04-25T11:39:24.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2aec1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 0x7f2aec149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0749a0 con 0x7f2aec154790 2024-04-25T11:39:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2aec0ca000 con 0x7f2ad800b680 2024-04-25T11:39:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af4440640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2aec074b20 con 0x7f2aec0c5070 2024-04-25T11:39:24.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af29dd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af29dd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36650/0 (socket says 172.21.15.136:36650) 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.057+0000 7f2af29dd640 1 -- 172.21.15.136:0/3557658622 learned_addr learned my addr 172.21.15.136:0/3557658622 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af19db640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 0x7f2aec149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 -- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 msgr2=0x7f2aec149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 0x7f2aec149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 -- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 msgr2=0x7f2aec1535b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2aec1535b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 -- 172.21.15.136:0/3557658622 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aec144b10 con 0x7f2aec154790 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af19db640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 0x7f2aec149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af29dd640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2ae401dd80 tx=0x7f2ad8003740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4004b70 con 0x7f2aec154790 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aec149f90 con 0x7f2aec154790 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aec155380 con 0x7f2aec154790 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae4004d10 con 0x7f2aec154790 2024-04-25T11:39:24.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.061+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae40ac9d0 con 0x7f2aec154790 2024-04-25T11:39:24.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.065+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2ae40ac420 con 0x7f2aec154790 2024-04-25T11:39:24.053 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.069+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad40052a0 con 0x7f2aec154790 2024-04-25T11:39:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.069+0000 7f2ae2ffd640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 0x7f2ad0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.073+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f2ae4129680 con 0x7f2aec154790 2024-04-25T11:39:24.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.077+0000 7f2af21dc640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 0x7f2ad0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:24.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.077+0000 7f2af21dc640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 0x7f2ad0071910 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0660c0 tx=0x7f2adc006b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:24.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.081+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2aec144b10 con 0x7f2aec154790 2024-04-25T11:39:24.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:39:24.525 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:39:24.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.685+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ad4005590 con 0x7f2aec154790 2024-04-25T11:39:24.668 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:24.668 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:24.008664+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5408,5449,5517,6246]},"id":1}]} 2024-04-25T11:39:24.668 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 80 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.685+0000 7f2ae2ffd640 1 -- 172.21.15.136:0/3557658622 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+6101 (secure 0 0 0) 0x7f2ae40a5e20 con 0x7f2aec154790 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 msgr2=0x7f2ad0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 0x7f2ad0071910 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0660c0 tx=0x7f2adc006b90 comp rx=0 tx=0).stop 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 msgr2=0x7f2aec1fd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2ae401dd80 tx=0x7f2ad8003740 comp rx=0 tx=0).stop 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 shutdown_connections 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2aec154790 0x7f2aec1fd740 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2ad006f400 0x7f2ad0071910 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2aec0c5070 0x7f2aec149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 --2- 172.21.15.136:0/3557658622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ad800b680 0x7f2aec1535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:24.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 >> 172.21.15.136:0/3557658622 conn(0x7f2aec0ca5a0 msgr2=0x7f2aec06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:24.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 shutdown_connections 2024-04-25T11:39:24.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:24.689+0000 7f2af4440640 1 -- 172.21.15.136:0/3557658622 wait complete. 2024-04-25T11:39:24.740 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-25T11:39:24.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:39:24.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:25.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.125+0000 7f0c5cb91640 1 Processor -- start 2024-04-25T11:39:25.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.125+0000 7f0c5cb91640 1 -- start start 2024-04-25T11:39:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.137+0000 7f0c5cb91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.137+0000 7f0c5cb91640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0c58073970 0x7f0c5806fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.137+0000 7f0c5cb91640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5816d050 con 0x7f0c58166390 2024-04-25T11:39:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.137+0000 7f0c5cb91640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0c581667a0 con 0x7f0c58073970 2024-04-25T11:39:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.137+0000 7f0c5cb91640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5808d2b0 con 0x7f0c5816cc40 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:47102/0 (socket says 172.21.15.136:47102) 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 -- 172.21.15.136:0/2383570832 learned_addr learned my addr 172.21.15.136:0/2383570832 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c56d9d640 1 --2- 172.21.15.136:0/2383570832 >> v2:172.21.15.157:3300/0 conn(0x7f0c58073970 0x7f0c5806fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 -- 172.21.15.136:0/2383570832 >> v2:172.21.15.157:3300/0 conn(0x7f0c58073970 msgr2=0x7f0c5806fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 --2- 172.21.15.136:0/2383570832 >> v2:172.21.15.157:3300/0 conn(0x7f0c58073970 0x7f0c5806fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 -- 172.21.15.136:0/2383570832 >> v1:172.21.15.136:6789/0 conn(0x7f0c58166390 legacy=0x7f0c58073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.141+0000 7f0c5659c640 1 -- 172.21.15.136:0/2383570832 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5808d560 con 0x7f0c5816cc40 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.145+0000 7f0c5659c640 1 --2- 172.21.15.136:0/2383570832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c580720d0 tx=0x7f0c480275d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66fe677f7481db3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.145+0000 7f0c5559a640 1 -- 172.21.15.136:0/2383570832 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480bd070 con 0x7f0c5816cc40 2024-04-25T11:39:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.145+0000 7f0c5559a640 1 -- 172.21.15.136:0/2383570832 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c480b4050 con 0x7f0c5816cc40 2024-04-25T11:39:25.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.145+0000 7f0c5cb91640 1 -- 172.21.15.136:0/2383570832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 msgr2=0x7f0c58165e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:25.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.145+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/2383570832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c580720d0 tx=0x7f0c480275d0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.149+0000 7f0c5cb91640 1 -- 172.21.15.136:0/2383570832 shutdown_connections 2024-04-25T11:39:25.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.149+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/2383570832 >> v2:172.21.15.157:3300/0 conn(0x7f0c58073970 0x7f0c5806fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.149+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/2383570832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c5816cc40 0x7f0c58165e50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.149+0000 7f0c5cb91640 1 -- 172.21.15.136:0/2383570832 >> 172.21.15.136:0/2383570832 conn(0x7f0c580c9d50 msgr2=0x7f0c580ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:25.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.161+0000 7f0c5cb91640 1 -- 172.21.15.136:0/2383570832 shutdown_connections 2024-04-25T11:39:25.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.177+0000 7f0c5cb91640 1 -- 172.21.15.136:0/2383570832 wait complete. 2024-04-25T11:39:25.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.181+0000 7f0c5cb91640 1 Processor -- start 2024-04-25T11:39:25.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.181+0000 7f0c5cb91640 1 -- start start 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 0x7f0c58168120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5816cc40 0x7f0c582017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0c58072a60 con 0x7f0c58073970 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5808db20 con 0x7f0c5816cc40 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5cb91640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0c58072d40 con 0x7f0c58166390 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c55d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 0x7f0c58168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c55d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 0x7f0c58168120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:47116/0 (socket says 172.21.15.136:47116) 2024-04-25T11:39:25.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36020/0 (socket says 172.21.15.136:36020) 2024-04-25T11:39:25.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c55d9b640 1 -- 172.21.15.136:0/475082342 learned_addr learned my addr 172.21.15.136:0/475082342 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:25.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 -- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 msgr2=0x7f0c58168120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:25.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 0x7f0c58168120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 -- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5816cc40 msgr2=0x7f0c582017f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5816cc40 0x7f0c582017f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 -- 172.21.15.136:0/475082342 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c58168660 con 0x7f0c58073970 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c5659c640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0c5808d430 tx=0x7f0c48024370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480bd070 con 0x7f0c58073970 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.185+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4800ae50 con 0x7f0c58073970 2024-04-25T11:39:25.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.189+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c480c13b0 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.193+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c58204f00 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.193+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c582054c0 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.193+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5806a350 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.193+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0c480c0070 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.197+0000 7f0c46ffd640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 0x7f0c38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.197+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f0c4812cab0 con 0x7f0c58073970 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.197+0000 7f0c55d9b640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 0x7f0c38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.197+0000 7f0c55d9b640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 0x7f0c38071910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0c58203ce0 tx=0x7f0c5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:25.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.221+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0c480c8020 con 0x7f0c58073970 2024-04-25T11:39:25.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.861+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c58076170 con 0x7f0c58073970 2024-04-25T11:39:25.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.865+0000 7f0c46ffd640 1 -- 172.21.15.136:0/475082342 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+6101 (secure 0 0 0) 0x7f0c48021130 con 0x7f0c58073970 2024-04-25T11:39:25.847 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:25.847 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:24.008664+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5408,5449,5517,6246]},"id":1}]} 2024-04-25T11:39:25.847 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 80 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 msgr2=0x7f0c38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 0x7f0c38071910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0c58203ce0 tx=0x7f0c5000b040 comp rx=0 tx=0).stop 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 msgr2=0x7f0c58167620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0c5808d430 tx=0x7f0c48024370 comp rx=0 tx=0).stop 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 shutdown_connections 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0c5816cc40 0x7f0c582017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0c3806f400 0x7f0c38071910 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0c58166390 0x7f0c58168120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 --2- 172.21.15.136:0/475082342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c58073970 0x7f0c58167620 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 >> 172.21.15.136:0/475082342 conn(0x7f0c580c9d50 msgr2=0x7f0c5806e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 shutdown_connections 2024-04-25T11:39:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:25.869+0000 7f0c5cb91640 1 -- 172.21.15.136:0/475082342 wait complete. 2024-04-25T11:39:25.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 80, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5408": { "addr": "172.21.15.157:6837/1501623433", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 1501623433, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 1501623433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5408, "incarnation": 74, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:39:24.008664+0000", "qdb_cluster": [ 5408, 5449, 5517, 6246 ], "qdb_leader": 5408, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5408, "mds_1": 5449, "mds_2": 5517, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:39:27.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:28.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 Processor -- start 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 -- start start 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8c06dff0 con 0x7f2e8c0c8f90 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8c0c7a10 con 0x7f2e8c0c5930 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8b59e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8c0c94e0 con 0x7f2e8c152e10 2024-04-25T11:39:28.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:47118/0 (socket says 172.21.15.136:47118) 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 -- 172.21.15.136:0/2562424532 learned_addr learned my addr 172.21.15.136:0/2562424532 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 -- 172.21.15.136:0/2562424532 >> v1:172.21.15.157:6789/0 conn(0x7f2e8c0c5930 legacy=0x7f2e8c0c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 -- 172.21.15.136:0/2562424532 >> v1:172.21.15.136:6789/0 conn(0x7f2e8c0c8f90 legacy=0x7f2e8c0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.389+0000 7f2e8a59c640 1 -- 172.21.15.136:0/2562424532 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c0c7cf0 con 0x7f2e8c152e10 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.393+0000 7f2e8a59c640 1 --2- 172.21.15.136:0/2562424532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c06f730 tx=0x7f2e8401c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d41ee6c8210a77e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.393+0000 7f2e8959a640 1 -- 172.21.15.136:0/2562424532 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840a5700 con 0x7f2e8c152e10 2024-04-25T11:39:28.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.393+0000 7f2e8959a640 1 -- 172.21.15.136:0/2562424532 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e840a58a0 con 0x7f2e8c152e10 2024-04-25T11:39:28.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.397+0000 7f2e8b59e640 1 -- 172.21.15.136:0/2562424532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 msgr2=0x7f2e8c0c8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:28.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.397+0000 7f2e8b59e640 1 --2- 172.21.15.136:0/2562424532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c06f730 tx=0x7f2e8401c8a0 comp rx=0 tx=0).stop 2024-04-25T11:39:28.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.401+0000 7f2e8b59e640 1 -- 172.21.15.136:0/2562424532 shutdown_connections 2024-04-25T11:39:28.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.401+0000 7f2e8b59e640 1 --2- 172.21.15.136:0/2562424532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c152e10 0x7f2e8c0c8a50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:28.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.401+0000 7f2e8b59e640 1 -- 172.21.15.136:0/2562424532 >> 172.21.15.136:0/2562424532 conn(0x7f2e8c073f30 msgr2=0x7f2e8c0714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:28.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.401+0000 7f2e8b59e640 1 -- 172.21.15.136:0/2562424532 shutdown_connections 2024-04-25T11:39:28.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.401+0000 7f2e8b59e640 1 -- 172.21.15.136:0/2562424532 wait complete. 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.405+0000 7f2e8b59e640 1 Processor -- start 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.405+0000 7f2e8b59e640 1 -- start start 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c0c8f90 0x7f2e8c144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 0x7f2e8c1453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c0ce6f0 con 0x7f2e8c152e10 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c0c82b0 con 0x7f2e8c0c5930 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8c0ce870 con 0x7f2e8c0c8f90 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:42128/0 (socket says 172.21.15.136:42128) 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 -- 172.21.15.136:0/3290093733 learned_addr learned my addr 172.21.15.136:0/3290093733 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8ad9d640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 0x7f2e8c1453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:28.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e89d9b640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c0c8f90 0x7f2e8c144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 -- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c0c8f90 msgr2=0x7f2e8c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c0c8f90 0x7f2e8c144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 -- 172.21.15.136:0/3290093733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 msgr2=0x7f2e8c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 0x7f2e8c1453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 -- 172.21.15.136:0/3290093733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c1562f0 con 0x7f2e8c0c5930 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8ad9d640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 0x7f2e8c1453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8a59c640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e8401d140 tx=0x7f2e84019640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840b8440 con 0x7f2e8c0c5930 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 -- 172.21.15.136:0/3290093733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c1565d0 con 0x7f2e8c0c5930 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.409+0000 7f2e8b59e640 1 -- 172.21.15.136:0/3290093733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8c154730 con 0x7f2e8c0c5930 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e840a5e50 con 0x7f2e8c0c5930 2024-04-25T11:39:28.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840b8800 con 0x7f2e8c0c5930 2024-04-25T11:39:28.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2e840c1020 con 0x7f2e8c0c5930 2024-04-25T11:39:28.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e7affd640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 0x7f2e6c0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:28.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f2e8412a580 con 0x7f2e8c0c5930 2024-04-25T11:39:28.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.413+0000 7f2e89d9b640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 0x7f2e6c0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:28.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.417+0000 7f2e89d9b640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 0x7f2e6c0719e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c156030 tx=0x7f2e7c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:28.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.421+0000 7f2e8b59e640 1 -- 172.21.15.136:0/3290093733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e700052a0 con 0x7f2e8c0c5930 2024-04-25T11:39:28.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:28.429+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2e840c8020 con 0x7f2e8c0c5930 2024-04-25T11:39:29.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.164+0000 7f2e8b59e640 1 -- 172.21.15.136:0/3290093733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e70005590 con 0x7f2e8c0c5930 2024-04-25T11:39:29.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.164+0000 7f2e7affd640 1 -- 172.21.15.136:0/3290093733 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6103 (secure 0 0 0) 0x7f2e84016560 con 0x7f2e8c0c5930 2024-04-25T11:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:29.076861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:resolve","state_seq":13,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408]},"id":1}]} 2024-04-25T11:39:29.148 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 81 2024-04-25T11:39:29.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 msgr2=0x7f2e6c0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:29.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 0x7f2e6c0719e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c156030 tx=0x7f2e7c009060 comp rx=0 tx=0).stop 2024-04-25T11:39:29.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 msgr2=0x7f2e8c154c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e8401d140 tx=0x7f2e84019640 comp rx=0 tx=0).stop 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 shutdown_connections 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c152e10 0x7f2e8c1453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2e6c06f4d0 0x7f2e6c0719e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e8c0c8f90 0x7f2e8c144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:29.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 --2- 172.21.15.136:0/3290093733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e8c0c5930 0x7f2e8c154c50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:29.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 >> 172.21.15.136:0/3290093733 conn(0x7f2e8c073f30 msgr2=0x7f2e8c0ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:29.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 shutdown_connections 2024-04-25T11:39:29.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:29.168+0000 7f2e78ff9640 1 -- 172.21.15.136:0/3290093733 wait complete. 2024-04-25T11:39:29.223 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:39:31.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:31.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.676+0000 7f7a093e9640 1 Processor -- start 2024-04-25T11:39:31.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.680+0000 7f7a093e9640 1 -- start start 2024-04-25T11:39:31.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a093e9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a040c3010 0x7f7a0406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a093e9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 0x7f7a040af640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a093e9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0406ea50 con 0x7f7a040afb80 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a027fc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 0x7f7a040af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a02ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a040c3010 0x7f7a0406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a02ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a040c3010 0x7f7a0406e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36038/0 (socket says 172.21.15.136:36038) 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a02ffd640 1 -- 172.21.15.136:0/2580391356 learned_addr learned my addr 172.21.15.136:0/2580391356 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7a0406ecc0 con 0x7f7a040c3010 2024-04-25T11:39:31.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.688+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7a040c3640 con 0x7f7a04152360 2024-04-25T11:39:31.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.692+0000 7f7a02ffd640 1 -- 172.21.15.136:0/2580391356 >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 msgr2=0x7f7a040af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:31.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.692+0000 7f7a02ffd640 1 --2- 172.21.15.136:0/2580391356 >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 0x7f7a040af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.692+0000 7f7a02ffd640 1 -- 172.21.15.136:0/2580391356 >> v1:172.21.15.157:6789/0 conn(0x7f7a040afb80 legacy=0x7f7a04157ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:31.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.692+0000 7f7a02ffd640 1 -- 172.21.15.136:0/2580391356 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a040c3920 con 0x7f7a040c3010 2024-04-25T11:39:31.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.692+0000 7f7a027fc640 1 --2- 172.21.15.136:0/2580391356 >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 0x7f7a040af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:39:31.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.696+0000 7f7a02ffd640 1 --2- 172.21.15.136:0/2580391356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a0406e510 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7a040ca990 tx=0x7f79fc01d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04d1caa4bf7f8c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:31.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.700+0000 7f7a01ffb640 1 -- 172.21.15.136:0/2580391356 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0b2070 con 0x7f7a040c3010 2024-04-25T11:39:31.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.700+0000 7f7a01ffb640 1 -- 172.21.15.136:0/2580391356 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79fc0ad670 con 0x7f7a040c3010 2024-04-25T11:39:31.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.700+0000 7f7a01ffb640 1 -- 172.21.15.136:0/2580391356 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc0ada40 con 0x7f7a040c3010 2024-04-25T11:39:31.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.700+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 msgr2=0x7f7a0406e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:31.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.700+0000 7f7a093e9640 1 --2- 172.21.15.136:0/2580391356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a0406e510 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7a040ca990 tx=0x7f79fc01d4f0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.712+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 shutdown_connections 2024-04-25T11:39:31.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.712+0000 7f7a093e9640 1 --2- 172.21.15.136:0/2580391356 >> v2:172.21.15.157:3300/0 conn(0x7f7a04152360 0x7f7a040af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.712+0000 7f7a093e9640 1 --2- 172.21.15.136:0/2580391356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a0406e510 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.712+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 >> 172.21.15.136:0/2580391356 conn(0x7f7a040cdae0 msgr2=0x7f7a040c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:31.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.716+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 shutdown_connections 2024-04-25T11:39:31.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.716+0000 7f7a093e9640 1 -- 172.21.15.136:0/2580391356 wait complete. 2024-04-25T11:39:31.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 Processor -- start 2024-04-25T11:39:31.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 -- start start 2024-04-25T11:39:31.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a040afb80 0x7f7a0414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a04145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.720+0000 7f7a093e9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04166420 con 0x7f7a040c3010 2024-04-25T11:39:31.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a093e9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041662a0 con 0x7f7a04152360 2024-04-25T11:39:31.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a093e9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a041665a0 con 0x7f7a040afb80 2024-04-25T11:39:31.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a02ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a040afb80 0x7f7a0414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:31.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:31.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:42140/0 (socket says 172.21.15.136:42140) 2024-04-25T11:39:31.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 -- 172.21.15.136:0/3544253035 learned_addr learned my addr 172.21.15.136:0/3544253035 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:31.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 -- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a040afb80 msgr2=0x7f7a0414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a040afb80 0x7f7a0414fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 -- 172.21.15.136:0/3544253035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 msgr2=0x7f7a04145be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a04145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 -- 172.21.15.136:0/3544253035 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a04150170 con 0x7f7a04152360 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a037fe640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f79f400b440 tx=0x7f79f400b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011830 con 0x7f7a04152360 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a040b0c80 con 0x7f7a04152360 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.724+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a040b1240 con 0x7f7a04152360 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f4016070 con 0x7f7a04152360 2024-04-25T11:39:31.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011c30 con 0x7f7a04152360 2024-04-25T11:39:31.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f79f401b460 con 0x7f7a04152360 2024-04-25T11:39:31.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e00052a0 con 0x7f7a04152360 2024-04-25T11:39:31.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f79f37fe640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 0x7f79e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:31.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.728+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f79f408f5f0 con 0x7f7a04152360 2024-04-25T11:39:31.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.732+0000 7f7a02ffd640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 0x7f79e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:31.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.732+0000 7f7a02ffd640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 0x7f79e8071910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0af000 tx=0x7f79fc0a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:31.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:31.740+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f79f405bd50 con 0x7f7a04152360 2024-04-25T11:39:32.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.380+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79e0005590 con 0x7f7a04152360 2024-04-25T11:39:32.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.380+0000 7f79f37fe640 1 -- 172.21.15.136:0/3544253035 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6103 (secure 0 0 0) 0x7f79f405b650 con 0x7f7a04152360 2024-04-25T11:39:32.366 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:32.366 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:29.076861+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:resolve","state_seq":13,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408]},"id":1}]} 2024-04-25T11:39:32.366 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 81 2024-04-25T11:39:32.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 msgr2=0x7f79e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:32.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 0x7f79e8071910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0af000 tx=0x7f79fc0a8000 comp rx=0 tx=0).stop 2024-04-25T11:39:32.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 msgr2=0x7f7a04164f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:32.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f79f400b440 tx=0x7f79f400b900 comp rx=0 tx=0).stop 2024-04-25T11:39:32.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 shutdown_connections 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a04152360 0x7f7a04164f80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a040c3010 0x7f7a04145be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f79e806f400 0x7f79e8071910 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 --2- 172.21.15.136:0/3544253035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a040afb80 0x7f7a0414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 >> 172.21.15.136:0/3544253035 conn(0x7f7a040cdae0 msgr2=0x7f7a040cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 shutdown_connections 2024-04-25T11:39:32.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:32.396+0000 7f7a093e9640 1 -- 172.21.15.136:0/3544253035 wait complete. 2024-04-25T11:39:32.452 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:39:34.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:34.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.832+0000 7f417df5f640 1 Processor -- start 2024-04-25T11:39:34.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.832+0000 7f417df5f640 1 -- start start 2024-04-25T11:39:34.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417df5f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4178160da0 0x7f41780c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:34.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417df5f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41781611b0 con 0x7f4178160990 2024-04-25T11:39:34.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417df5f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417815a940 con 0x7f41780c8060 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417df5f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f41780c3560 con 0x7f4178160da0 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417d75e640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f41780c8060 0x7f417816abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:48998/0 (socket says 172.21.15.136:48998) 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f417d75e640 1 -- 172.21.15.136:0/3038141804 learned_addr learned my addr 172.21.15.136:0/3038141804 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f4177fff640 1 --2- 172.21.15.136:0/3038141804 >> v2:172.21.15.157:3300/0 conn(0x7f4178160da0 0x7f41780c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f4177fff640 1 -- 172.21.15.136:0/3038141804 >> v1:172.21.15.157:6789/0 conn(0x7f41780c8060 legacy=0x7f417816abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f4177fff640 1 -- 172.21.15.136:0/3038141804 >> v1:172.21.15.136:6789/0 conn(0x7f4178160990 legacy=0x7f41780c8860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:34.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.836+0000 7f4177fff640 1 -- 172.21.15.136:0/3038141804 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4178163c90 con 0x7f4178160da0 2024-04-25T11:39:34.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.840+0000 7f4177fff640 1 --2- 172.21.15.136:0/3038141804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160da0 0x7f41780c7b20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4168009740 tx=0x7f416801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1000e39214b5d045 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:34.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.844+0000 7f41777fe640 1 -- 172.21.15.136:0/3038141804 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41680b2070 con 0x7f4178160da0 2024-04-25T11:39:34.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.844+0000 7f41777fe640 1 -- 172.21.15.136:0/3038141804 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41680a9070 con 0x7f4178160da0 2024-04-25T11:39:34.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.844+0000 7f417df5f640 1 -- 172.21.15.136:0/3038141804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160da0 msgr2=0x7f41780c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:34.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.844+0000 7f417df5f640 1 --2- 172.21.15.136:0/3038141804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160da0 0x7f41780c7b20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4168009740 tx=0x7f416801c300 comp rx=0 tx=0).stop 2024-04-25T11:39:34.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.848+0000 7f417df5f640 1 -- 172.21.15.136:0/3038141804 shutdown_connections 2024-04-25T11:39:34.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.848+0000 7f417df5f640 1 --2- 172.21.15.136:0/3038141804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160da0 0x7f41780c7b20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:34.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.848+0000 7f417df5f640 1 -- 172.21.15.136:0/3038141804 >> 172.21.15.136:0/3038141804 conn(0x7f417806d100 msgr2=0x7f4178070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:34.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.848+0000 7f417df5f640 1 -- 172.21.15.136:0/3038141804 shutdown_connections 2024-04-25T11:39:34.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.848+0000 7f417df5f640 1 -- 172.21.15.136:0/3038141804 wait complete. 2024-04-25T11:39:34.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.852+0000 7f417df5f640 1 Processor -- start 2024-04-25T11:39:34.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.856+0000 7f417df5f640 1 -- start start 2024-04-25T11:39:34.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:34.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 0x7f41781622e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:34.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417cf5d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:34.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417cf5d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33524/0 (socket says 172.21.15.136:33524) 2024-04-25T11:39:34.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 0x7f41781462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4178164530 con 0x7f41780c8060 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4178164250 con 0x7f4178160990 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417df5f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4178160470 con 0x7f4178160da0 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f4177fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 0x7f41781622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f4177fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 0x7f41781622e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:43666/0 (socket says 172.21.15.136:43666) 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f4177fff640 1 -- 172.21.15.136:0/374540146 learned_addr learned my addr 172.21.15.136:0/374540146 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 0x7f41781462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:34.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 -- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 msgr2=0x7f41781622e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:34.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 0x7f41781622e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:34.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 -- 172.21.15.136:0/374540146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 msgr2=0x7f41780cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:34.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:34.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 -- 172.21.15.136:0/374540146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780cd640 con 0x7f4178160da0 2024-04-25T11:39:34.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417d75e640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 0x7f41781462a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f417000e390 tx=0x7f417000e850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:34.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.860+0000 7f417cf5d640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:34.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.864+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170019070 con 0x7f4178160da0 2024-04-25T11:39:34.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.864+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41780cc6a0 con 0x7f4178160da0 2024-04-25T11:39:34.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.864+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f417814cd40 con 0x7f4178160da0 2024-04-25T11:39:34.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.864+0000 7f4166ffd640 1 -- 172.21.15.136:0/374540146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41540052a0 con 0x7f4178160da0 2024-04-25T11:39:34.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4170009480 con 0x7f4178160da0 2024-04-25T11:39:34.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170009830 con 0x7f4178160da0 2024-04-25T11:39:34.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f417001c070 con 0x7f4178160da0 2024-04-25T11:39:34.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f41757fa640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 0x7f415c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:34.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f417cf5d640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 0x7f415c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f4170091f00 con 0x7f4178160da0 2024-04-25T11:39:34.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.868+0000 7f417cf5d640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 0x7f415c071a10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f416c006fd0 tx=0x7f416c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:34.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:34.876+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f417005e050 con 0x7f4178160da0 2024-04-25T11:39:35.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.540+0000 7f4166ffd640 1 -- 172.21.15.136:0/374540146 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4154005590 con 0x7f4178160da0 2024-04-25T11:39:35.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.540+0000 7f41757fa640 1 -- 172.21.15.136:0/374540146 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6105 (secure 0 0 0) 0x7f4170062350 con 0x7f4178160da0 2024-04-25T11:39:35.530 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:35.530 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:34.138230+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5408,5449,5517,6246]},"id":1}]} 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 82 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 msgr2=0x7f415c071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 0x7f415c071a10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f416c006fd0 tx=0x7f416c009060 comp rx=0 tx=0).stop 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 msgr2=0x7f41781462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 0x7f41781462a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f417000e390 tx=0x7f417000e850 comp rx=0 tx=0).stop 2024-04-25T11:39:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 shutdown_connections 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f415c06f500 0x7f415c071a10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4178160da0 0x7f41781462a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4178160990 0x7f41781622e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 --2- 172.21.15.136:0/374540146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41780c8060 0x7f41780cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 >> 172.21.15.136:0/374540146 conn(0x7f417806d100 msgr2=0x7f41781561c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 shutdown_connections 2024-04-25T11:39:35.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:35.548+0000 7f417df5f640 1 -- 172.21.15.136:0/374540146 wait complete. 2024-04-25T11:39:35.624 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:39:37.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.476+0000 7faf09a9e640 1 Processor -- start 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.480+0000 7faf09a9e640 1 -- start start 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf09a9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf09a9e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 0x7faf04158980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf09a9e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0415be80 con 0x7faf04158570 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf09a9e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faf04152600 con 0x7faf040c9e40 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf09a9e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf0415c3e0 con 0x7faf041521f0 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:41578/0 (socket says 172.21.15.136:41578) 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 -- 172.21.15.136:0/4199973589 learned_addr learned my addr 172.21.15.136:0/4199973589 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf0929d640 1 --2- 172.21.15.136:0/4199973589 >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 0x7faf04158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 -- 172.21.15.136:0/4199973589 >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 msgr2=0x7faf04158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 --2- 172.21.15.136:0/4199973589 >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 0x7faf04158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 -- 172.21.15.136:0/4199973589 >> v1:172.21.15.157:6789/0 conn(0x7faf04158570 legacy=0x7faf040c9730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf08a9c640 1 -- 172.21.15.136:0/4199973589 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0415c100 con 0x7faf041521f0 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.484+0000 7faf0929d640 1 --2- 172.21.15.136:0/4199973589 >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 0x7faf04158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf08a9c640 1 --2- 172.21.15.136:0/4199973589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faefc00d1e0 tx=0x7faefc01eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0fbf5ce17b4be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf037fe640 1 -- 172.21.15.136:0/4199973589 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0ad070 con 0x7faf041521f0 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf037fe640 1 -- 172.21.15.136:0/4199973589 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faefc0a7ad0 con 0x7faf041521f0 2024-04-25T11:39:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 -- 172.21.15.136:0/4199973589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 msgr2=0x7faf040c2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 --2- 172.21.15.136:0/4199973589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faefc00d1e0 tx=0x7faefc01eb30 comp rx=0 tx=0).stop 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 -- 172.21.15.136:0/4199973589 shutdown_connections 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 --2- 172.21.15.136:0/4199973589 >> v2:172.21.15.157:3300/0 conn(0x7faf040c9e40 0x7faf04158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 --2- 172.21.15.136:0/4199973589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf041521f0 0x7faf040c2470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 -- 172.21.15.136:0/4199973589 >> 172.21.15.136:0/4199973589 conn(0x7faf0406d100 msgr2=0x7faf04070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 -- 172.21.15.136:0/4199973589 shutdown_connections 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.488+0000 7faf09a9e640 1 -- 172.21.15.136:0/4199973589 wait complete. 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.492+0000 7faf09a9e640 1 Processor -- start 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.492+0000 7faf09a9e640 1 -- start start 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf041521f0 0x7faf041ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 0x7faf041fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf08a9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf08a9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33528/0 (socket says 172.21.15.136:33528) 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf08a9c640 1 -- 172.21.15.136:0/3468865812 learned_addr learned my addr 172.21.15.136:0/3468865812 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faf04165050 con 0x7faf040c9e40 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf040ca250 con 0x7faf041521f0 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf041651d0 con 0x7faf04158570 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf0929d640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 0x7faf041fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.496+0000 7faf03fff640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf041521f0 0x7faf041ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 -- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 msgr2=0x7faf041fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 0x7faf041fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 -- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf041521f0 msgr2=0x7faf041ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf041521f0 0x7faf041ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf040c4e20 con 0x7faf040c9e40 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf08a9c640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7faefc01d1e0 tx=0x7faefc01b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0c0070 con 0x7faf040c9e40 2024-04-25T11:39:41.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faefc0b1dd0 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0415add0 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc0ad040 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf04159f30 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.500+0000 7faf0929d640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 0x7faf041fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.504+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee40052a0 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.504+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faefc0c3070 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.504+0000 7faf017fa640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 0x7faef0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.504+0000 7faf03fff640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 0x7faef0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.504+0000 7faf03fff640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 0x7faef0071910 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faf041513c0 tx=0x7faef8009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.512+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7faefc12e060 con 0x7faf040c9e40 2024-04-25T11:39:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:41.512+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faf0415be80 con 0x7faf040c9e40 2024-04-25T11:39:42.108 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.124+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faee4005590 con 0x7faf040c9e40 2024-04-25T11:39:42.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.124+0000 7faf017fa640 1 -- 172.21.15.136:0/3468865812 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+6102 (secure 0 0 0) 0x7faefc0f5da0 con 0x7faf040c9e40 2024-04-25T11:39:42.109 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:42.109 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:40.022953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5449,5408]},"id":1}]} 2024-04-25T11:39:42.110 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 83 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 msgr2=0x7faef0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 0x7faef0071910 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faf041513c0 tx=0x7faef8009060 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 msgr2=0x7faf040c48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7faefc01d1e0 tx=0x7faefc01b8d0 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 shutdown_connections 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7faef006f400 0x7faef0071910 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf04158570 0x7faf041fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf041521f0 0x7faf041ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 --2- 172.21.15.136:0/3468865812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf040c9e40 0x7faf040c48e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 >> 172.21.15.136:0/3468865812 conn(0x7faf0406d100 msgr2=0x7faf04073440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:42.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.128+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 shutdown_connections 2024-04-25T11:39:42.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:42.132+0000 7faf09a9e640 1 -- 172.21.15.136:0/3468865812 wait complete. 2024-04-25T11:39:42.761 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:39:44.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:47.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.536+0000 7fea75fa9640 1 Processor -- start 2024-04-25T11:39:47.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.536+0000 7fea75fa9640 1 -- start start 2024-04-25T11:39:47.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea75fa9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea701524e0 0x7fea700c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea75fa9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7015c5e0 con 0x7fea7006da90 2024-04-25T11:39:47.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea75fa9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea70158f20 con 0x7fea7006d680 2024-04-25T11:39:47.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea75fa9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fea70158cd0 con 0x7fea701524e0 2024-04-25T11:39:47.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea701524e0 0x7fea700c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea701524e0 0x7fea700c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33726/0 (socket says 172.21.15.136:33726) 2024-04-25T11:39:47.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 -- 172.21.15.136:0/3662124606 learned_addr learned my addr 172.21.15.136:0/3662124606 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:47.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 -- 172.21.15.136:0/3662124606 >> v1:172.21.15.157:6789/0 conn(0x7fea7006d680 legacy=0x7fea7015b7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:47.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 -- 172.21.15.136:0/3662124606 >> v1:172.21.15.136:6789/0 conn(0x7fea7006da90 legacy=0x7fea7015bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:39:47.525 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.540+0000 7fea74fa7640 1 -- 172.21.15.136:0/3662124606 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea700c65c0 con 0x7fea701524e0 2024-04-25T11:39:47.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea74fa7640 1 --2- 172.21.15.136:0/3662124606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea701524e0 0x7fea700c7380 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fea64009ad0 tx=0x7fea6401ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cfb9ad780fcfc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea6f7fe640 1 -- 172.21.15.136:0/3662124606 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea640b5070 con 0x7fea701524e0 2024-04-25T11:39:47.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea6f7fe640 1 -- 172.21.15.136:0/3662124606 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea640a8c30 con 0x7fea701524e0 2024-04-25T11:39:47.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea6f7fe640 1 -- 172.21.15.136:0/3662124606 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea640b0440 con 0x7fea701524e0 2024-04-25T11:39:47.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 -- 172.21.15.136:0/3662124606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea701524e0 msgr2=0x7fea700c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3662124606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea701524e0 0x7fea700c7380 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fea64009ad0 tx=0x7fea6401ef30 comp rx=0 tx=0).stop 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 -- 172.21.15.136:0/3662124606 shutdown_connections 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3662124606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea701524e0 0x7fea700c7380 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 -- 172.21.15.136:0/3662124606 >> 172.21.15.136:0/3662124606 conn(0x7fea7006e6c0 msgr2=0x7fea7006ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 -- 172.21.15.136:0/3662124606 shutdown_connections 2024-04-25T11:39:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.544+0000 7fea75fa9640 1 -- 172.21.15.136:0/3662124606 wait complete. 2024-04-25T11:39:47.532 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.548+0000 7fea75fa9640 1 Processor -- start 2024-04-25T11:39:47.534 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.548+0000 7fea75fa9640 1 -- start start 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 0x7fea700c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 0x7fea701506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 0x7fea701402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fea700c6b80 con 0x7fea7006d680 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fea70159200 con 0x7fea7006da90 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fea700c6e60 con 0x7fea701f6fd0 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea6ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 0x7fea701506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea6ffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 0x7fea701506d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:53020/0 (socket says 172.21.15.136:53020) 2024-04-25T11:39:47.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea6ffff640 1 -- 172.21.15.136:0/3319476842 learned_addr learned my addr 172.21.15.136:0/3319476842 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:47.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 0x7fea701402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea74fa7640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 0x7fea700c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 -- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 msgr2=0x7fea701506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 0x7fea701506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 -- 172.21.15.136:0/3319476842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 msgr2=0x7fea700c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 0x7fea700c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 -- 172.21.15.136:0/3319476842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea701f6810 con 0x7fea701f6fd0 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea74fa7640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 0x7fea700c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:39:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea757a8640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 0x7fea701402e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea6800bdf0 tx=0x7fea68009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68019070 con 0x7fea701f6fd0 2024-04-25T11:39:47.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea700c8940 con 0x7fea701f6fd0 2024-04-25T11:39:47.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.560+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea701f6ab0 con 0x7fea701f6fd0 2024-04-25T11:39:47.547 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.564+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea68014440 con 0x7fea701f6fd0 2024-04-25T11:39:47.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.564+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68014830 con 0x7fea701f6fd0 2024-04-25T11:39:47.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.568+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea6801c070 con 0x7fea701f6fd0 2024-04-25T11:39:47.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.568+0000 7fea6d7fa640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 0x7fea540719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:47.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.568+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7fea6808f9e0 con 0x7fea701f6fd0 2024-04-25T11:39:47.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.568+0000 7fea74fa7640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 0x7fea540719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:47.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.568+0000 7fea74fa7640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 0x7fea540719e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fea640b2000 tx=0x7fea640aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:47.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.592+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea500052a0 con 0x7fea701f6fd0 2024-04-25T11:39:47.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:47.600+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fea6805baa0 con 0x7fea701f6fd0 2024-04-25T11:39:48.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.332+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea50005590 con 0x7fea701f6fd0 2024-04-25T11:39:48.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.336+0000 7fea6d7fa640 1 -- 172.21.15.136:0/3319476842 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+6107 (secure 0 0 0) 0x7fea6805fda0 con 0x7fea701f6fd0 2024-04-25T11:39:48.322 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:48.322 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[5408,5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:39:48.323 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 84 2024-04-25T11:39:48.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 msgr2=0x7fea540719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 0x7fea540719e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fea640b2000 tx=0x7fea640aa070 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 msgr2=0x7fea701402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 0x7fea701402e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea6800bdf0 tx=0x7fea68009ef0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 shutdown_connections 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fea5406f4d0 0x7fea540719e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fea701f6fd0 0x7fea701402e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fea7006da90 0x7fea701506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 --2- 172.21.15.136:0/3319476842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea7006d680 0x7fea700c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 >> 172.21.15.136:0/3319476842 conn(0x7fea7006e6c0 msgr2=0x7fea700ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 shutdown_connections 2024-04-25T11:39:48.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:48.348+0000 7fea75fa9640 1 -- 172.21.15.136:0/3319476842 wait complete. 2024-04-25T11:39:48.398 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=3, replacing gid=5385 2024-04-25T11:39:48.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2024-04-25T11:39:50.400 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-25T11:39:50.400 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T11:39:50.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T11:39:50.402 INFO:tasks.ceph.mds.a:Started 2024-04-25T11:39:50.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:39:51.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 Processor -- start 2024-04-25T11:39:51.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 -- start start 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51a006f910 0x7f51a00c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a0160270 con 0x7f51a00c3a90 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a00c92a0 con 0x7f51a00c29f0 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.200+0000 7f51a822c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f51a00c9820 con 0x7f51a006f910 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51a006f910 0x7f51a00c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51a006f910 0x7f51a00c3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:53024/0 (socket says 172.21.15.136:53024) 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 -- 172.21.15.136:0/353267135 learned_addr learned my addr 172.21.15.136:0/353267135 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 -- 172.21.15.136:0/353267135 >> v1:172.21.15.157:6789/0 conn(0x7f51a00c29f0 legacy=0x7f51a016ab10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 -- 172.21.15.136:0/353267135 >> v1:172.21.15.136:6789/0 conn(0x7f51a00c3a90 legacy=0x7f51a0073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.204+0000 7f51a5fc8640 1 -- 172.21.15.136:0/353267135 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a006ef20 con 0x7f51a006f910 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a5fc8640 1 --2- 172.21.15.136:0/353267135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a006f910 0x7f51a00c3550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f519801ae50 tx=0x7f519802d920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3d5c77c411a78d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a4fc6640 1 -- 172.21.15.136:0/353267135 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51980b6700 con 0x7f51a006f910 2024-04-25T11:39:51.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a4fc6640 1 -- 172.21.15.136:0/353267135 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51980b68a0 con 0x7f51a006f910 2024-04-25T11:39:51.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a822c640 1 -- 172.21.15.136:0/353267135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a006f910 msgr2=0x7f51a00c3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:51.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a822c640 1 --2- 172.21.15.136:0/353267135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a006f910 0x7f51a00c3550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f519801ae50 tx=0x7f519802d920 comp rx=0 tx=0).stop 2024-04-25T11:39:51.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a822c640 1 -- 172.21.15.136:0/353267135 shutdown_connections 2024-04-25T11:39:51.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a822c640 1 --2- 172.21.15.136:0/353267135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a006f910 0x7f51a00c3550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.212+0000 7f51a822c640 1 -- 172.21.15.136:0/353267135 >> 172.21.15.136:0/353267135 conn(0x7f51a00cdae0 msgr2=0x7f51a00c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:51.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.216+0000 7f51a822c640 1 -- 172.21.15.136:0/353267135 shutdown_connections 2024-04-25T11:39:51.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.216+0000 7f51a822c640 1 -- 172.21.15.136:0/353267135 wait complete. 2024-04-25T11:39:51.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.216+0000 7f51a822c640 1 Processor -- start 2024-04-25T11:39:51.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.216+0000 7f51a822c640 1 -- start start 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51a00c29f0 0x7f51a00cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a00c3a90 0x7f51a015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51a00743b0 con 0x7f51a006f910 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51a0074230 con 0x7f51a00c3a90 2024-04-25T11:39:51.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a822c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51a0074fd0 con 0x7f51a00c29f0 2024-04-25T11:39:51.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:51.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33736/0 (socket says 172.21.15.136:33736) 2024-04-25T11:39:51.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 -- 172.21.15.136:0/1683877992 learned_addr learned my addr 172.21.15.136:0/1683877992 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a67c9640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a00c3a90 0x7f51a015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 -- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51a00c29f0 msgr2=0x7f51a00cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51a00c29f0 0x7f51a00cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 -- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a00c3a90 msgr2=0x7f51a015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a00c3a90 0x7f51a015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 -- 172.21.15.136:0/1683877992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a01f14c0 con 0x7f51a006f910 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51a5fc8640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f51980c0040 tx=0x7f519802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51980b6700 con 0x7f51a006f910 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51980cf070 con 0x7f51a006f910 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.220+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51980c96b0 con 0x7f51a006f910 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.224+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a01500a0 con 0x7f51a006f910 2024-04-25T11:39:51.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.224+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a0150660 con 0x7f51a006f910 2024-04-25T11:39:51.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.224+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a006a2e0 con 0x7f51a006f910 2024-04-25T11:39:51.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.228+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51980b68a0 con 0x7f51a006f910 2024-04-25T11:39:51.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.228+0000 7f51967fc640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 0x7f51840719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:39:51.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.228+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f519813bdb0 con 0x7f51a006f910 2024-04-25T11:39:51.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.228+0000 7f51a57c7640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 0x7f51840719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:39:51.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.240+0000 7f51a57c7640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 0x7f51840719e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f51a0164850 tx=0x7f5190006bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:39:51.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.244+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f51980275a0 con 0x7f51a006f910 2024-04-25T11:39:51.311 INFO:tasks.ceph.mds.a.smithi136.stdout:starting mds.a at 2024-04-25T11:39:51.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.944+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51a00c9080 con 0x7f51a006f910 2024-04-25T11:39:51.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.944+0000 7f51967fc640 1 -- 172.21.15.136:0/1683877992 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7f51981084c0 con 0x7f51a006f910 2024-04-25T11:39:51.931 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:39:51.932 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:39:51.932 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:39:51.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.948+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 msgr2=0x7f51840719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:51.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.948+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 0x7f51840719e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f51a0164850 tx=0x7f5190006bd0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.948+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 msgr2=0x7f51a01f0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:39:51.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.948+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f51980c0040 tx=0x7f519802a6c0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 shutdown_connections 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51a00c3a90 0x7f51a015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f518406f4d0 0x7f51840719e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51a00c29f0 0x7f51a00cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 --2- 172.21.15.136:0/1683877992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51a006f910 0x7f51a01f0f80 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 >> 172.21.15.136:0/1683877992 conn(0x7f51a00cdae0 msgr2=0x7f51a006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:39:51.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 shutdown_connections 2024-04-25T11:39:51.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:39:51.960+0000 7f51a822c640 1 -- 172.21.15.136:0/1683877992 wait complete. 2024-04-25T11:39:52.028 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-25T11:39:52.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2024-04-25T11:39:54.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:39:54.827 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:40:25.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:40:25.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:40:56.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:40:56.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:41:26.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:41:26.380 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:41:43.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:43.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 Processor -- start 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 -- start start 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1d2016cc40 0x7f1d20165e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2016d050 con 0x7f1d20166390 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1d201667a0 con 0x7f1d2016cc40 2024-04-25T11:41:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.912+0000 7f1d28191640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2008d2b0 con 0x7f1d20073970 2024-04-25T11:41:43.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:45394/0 (socket says 172.21.15.136:45394) 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 -- 172.21.15.136:0/3827320434 learned_addr learned my addr 172.21.15.136:0/3827320434 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 -- 172.21.15.136:0/3827320434 >> v2:172.21.15.136:3300/0 conn(0x7f1d2016cc40 msgr2=0x7f1d20165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 --2- 172.21.15.136:0/3827320434 >> v2:172.21.15.136:3300/0 conn(0x7f1d2016cc40 0x7f1d20165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 -- 172.21.15.136:0/3827320434 >> v1:172.21.15.136:6789/0 conn(0x7f1d20166390 legacy=0x7f1d20073260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 -- 172.21.15.136:0/3827320434 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2008d560 con 0x7f1d20073970 2024-04-25T11:41:43.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.924+0000 7f1d2672e640 1 --2- 172.21.15.136:0/3827320434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1d18006140 tx=0x7f1d180279c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2423d9c4cacd28ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:43.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.928+0000 7f1d24f2b640 1 -- 172.21.15.136:0/3827320434 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180b7070 con 0x7f1d20073970 2024-04-25T11:41:43.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.928+0000 7f1d24f2b640 1 -- 172.21.15.136:0/3827320434 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d180b1790 con 0x7f1d20073970 2024-04-25T11:41:43.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.928+0000 7f1d24f2b640 1 -- 172.21.15.136:0/3827320434 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180b1b60 con 0x7f1d20073970 2024-04-25T11:41:43.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 -- 172.21.15.136:0/3827320434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 msgr2=0x7f1d2006fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:43.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 --2- 172.21.15.136:0/3827320434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1d18006140 tx=0x7f1d180279c0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 -- 172.21.15.136:0/3827320434 shutdown_connections 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 --2- 172.21.15.136:0/3827320434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d2006fec0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 --2- 172.21.15.136:0/3827320434 >> v2:172.21.15.136:3300/0 conn(0x7f1d2016cc40 0x7f1d20165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 -- 172.21.15.136:0/3827320434 >> 172.21.15.136:0/3827320434 conn(0x7f1d200c9d50 msgr2=0x7f1d200ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 -- 172.21.15.136:0/3827320434 shutdown_connections 2024-04-25T11:41:43.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.932+0000 7f1d28191640 1 -- 172.21.15.136:0/3827320434 wait complete. 2024-04-25T11:41:43.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.936+0000 7f1d28191640 1 Processor -- start 2024-04-25T11:41:43.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 -- start start 2024-04-25T11:41:43.925 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d201f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 0x7f1d200ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20072a60 con 0x7f1d20166390 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2008db20 con 0x7f1d2016cc40 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d28191640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1d20072d40 con 0x7f1d20073970 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2572c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 0x7f1d200ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:43.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2572c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 0x7f1d200ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36402/0 (socket says 172.21.15.136:36402) 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:45194/0 (socket says 172.21.15.136:45194) 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2572c640 1 -- 172.21.15.136:0/2453655712 learned_addr learned my addr 172.21.15.136:0/2453655712 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 -- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 msgr2=0x7f1d201f4790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d201f4790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 -- 172.21.15.136:0/2453655712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 msgr2=0x7f1d200ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 0x7f1d200ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 -- 172.21.15.136:0/2453655712 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d201ff950 con 0x7f1d2016cc40 2024-04-25T11:41:43.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.940+0000 7f1d2672e640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d180b9000 tx=0x7f1d18024760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:43.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.968+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180b7040 con 0x7f1d2016cc40 2024-04-25T11:41:43.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.968+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d201f3a80 con 0x7f1d2016cc40 2024-04-25T11:41:43.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.968+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201f4000 con 0x7f1d2016cc40 2024-04-25T11:41:43.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.968+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1801a5a0 con 0x7f1d2016cc40 2024-04-25T11:41:43.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.968+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d180bb640 con 0x7f1d2016cc40 2024-04-25T11:41:43.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.972+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d080052a0 con 0x7f1d2016cc40 2024-04-25T11:41:43.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.972+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1d180bb7e0 con 0x7f1d2016cc40 2024-04-25T11:41:43.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.972+0000 7f1d167fc640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 0x7f1d040719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:43.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.972+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f1d1813e920 con 0x7f1d2016cc40 2024-04-25T11:41:43.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.972+0000 7f1d25f2d640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 0x7f1d040719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:43.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.976+0000 7f1d25f2d640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 0x7f1d040719e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009670 tx=0x7f1d10006b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:43.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:43.980+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1d180db070 con 0x7f1d2016cc40 2024-04-25T11:41:44.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.696+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d08005590 con 0x7f1d2016cc40 2024-04-25T11:41:44.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.700+0000 7f1d167fc640 1 -- 172.21.15.136:0/2453655712 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7f1d18021520 con 0x7f1d2016cc40 2024-04-25T11:41:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:44.688 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:41:44.688 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 msgr2=0x7f1d040719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 0x7f1d040719e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009670 tx=0x7f1d10006b90 comp rx=0 tx=0).stop 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 msgr2=0x7f1d201ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d180b9000 tx=0x7f1d18024760 comp rx=0 tx=0).stop 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 shutdown_connections 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d2016cc40 0x7f1d201ff410 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d20166390 0x7f1d200ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:44.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1d0406f4d0 0x7f1d040719e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 --2- 172.21.15.136:0/2453655712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d20073970 0x7f1d201f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 >> 172.21.15.136:0/2453655712 conn(0x7f1d200c9d50 msgr2=0x7f1d200c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 shutdown_connections 2024-04-25T11:41:44.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:44.708+0000 7f1d28191640 1 -- 172.21.15.136:0/2453655712 wait complete. 2024-04-25T11:41:44.834 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2024-04-25T11:41:44.834 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T11:41:44.834 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:41:44.855 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T11:41:44.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:45.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.592+0000 7f15ff271640 1 Processor -- start 2024-04-25T11:41:45.589 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.604+0000 7f15ff271640 1 -- start start 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15ff271640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15ff271640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f815c860 con 0x7f15f81561e0 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15ff271640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f815cae0 con 0x7f15f80c9460 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15ff271640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f15f81565f0 con 0x7f15f80ca110 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58726/0 (socket says 172.21.15.136:58726) 2024-04-25T11:41:45.608 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fc80c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f15f80c9460 0x7f15f80c9a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:35866/0 (socket says 172.21.15.136:35866) 2024-04-25T11:41:45.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 -- 172.21.15.136:0/107773718 learned_addr learned my addr 172.21.15.136:0/107773718 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:45.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 -- 172.21.15.136:0/107773718 >> v1:172.21.15.157:6789/0 conn(0x7f15f80c9460 legacy=0x7f15f80c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:45.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 -- 172.21.15.136:0/107773718 >> v1:172.21.15.136:6789/0 conn(0x7f15f81561e0 legacy=0x7f15f80c8d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:45.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.624+0000 7f15fd80e640 1 -- 172.21.15.136:0/107773718 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f80b6d40 con 0x7f15f80ca110 2024-04-25T11:41:45.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.628+0000 7f15fd80e640 1 --2- 172.21.15.136:0/107773718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009e80 tx=0x7f15f001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=389e86aefcbf38f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:45.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.632+0000 7f15effff640 1 -- 172.21.15.136:0/107773718 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f00a5600 con 0x7f15f80ca110 2024-04-25T11:41:45.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.632+0000 7f15effff640 1 -- 172.21.15.136:0/107773718 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f00a57a0 con 0x7f15f80ca110 2024-04-25T11:41:45.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.632+0000 7f15effff640 1 -- 172.21.15.136:0/107773718 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f00a5b70 con 0x7f15f80ca110 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.632+0000 7f15ff271640 1 -- 172.21.15.136:0/107773718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 msgr2=0x7f15f81676e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.632+0000 7f15ff271640 1 --2- 172.21.15.136:0/107773718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009e80 tx=0x7f15f001c830 comp rx=0 tx=0).stop 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.636+0000 7f15ff271640 1 -- 172.21.15.136:0/107773718 shutdown_connections 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.636+0000 7f15ff271640 1 --2- 172.21.15.136:0/107773718 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f81676e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.636+0000 7f15ff271640 1 -- 172.21.15.136:0/107773718 >> 172.21.15.136:0/107773718 conn(0x7f15f80d1290 msgr2=0x7f15f80d16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:45.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.636+0000 7f15ff271640 1 -- 172.21.15.136:0/107773718 shutdown_connections 2024-04-25T11:41:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.636+0000 7f15ff271640 1 -- 172.21.15.136:0/107773718 wait complete. 2024-04-25T11:41:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.640+0000 7f15ff271640 1 Processor -- start 2024-04-25T11:41:45.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.640+0000 7f15ff271640 1 -- start start 2024-04-25T11:41:45.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48836/0 (socket says 172.21.15.136:48836) 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 -- 172.21.15.136:0/339176433 learned_addr learned my addr 172.21.15.136:0/339176433 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15f81561e0 0x7f15f81ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f15f80c88c0 con 0x7f15f80c9460 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f15f815cdc0 con 0x7f15f81561e0 2024-04-25T11:41:45.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f15f816a440 con 0x7f15f80ca110 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 -- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 msgr2=0x7f15f80c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 -- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15f81561e0 msgr2=0x7f15f81ff7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15f81561e0 0x7f15f81ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f80739d0 con 0x7f15f80c9460 2024-04-25T11:41:45.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.644+0000 7f15fd00d640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f15e800b420 tx=0x7f15e800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:45.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.652+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8015020 con 0x7f15f80c9460 2024-04-25T11:41:45.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.652+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e8011850 con 0x7f15f80c9460 2024-04-25T11:41:45.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.652+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8011c40 con 0x7f15f80c9460 2024-04-25T11:41:45.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.652+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f8205c80 con 0x7f15f80c9460 2024-04-25T11:41:45.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.652+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f8155630 con 0x7f15f80c9460 2024-04-25T11:41:45.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.656+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f15e8018020 con 0x7f15f80c9460 2024-04-25T11:41:45.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.656+0000 7f15edffb640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 0x7f15dc0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:45.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.656+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7f15e8093060 con 0x7f15f80c9460 2024-04-25T11:41:45.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.660+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15d80052a0 con 0x7f15f80c9460 2024-04-25T11:41:45.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.660+0000 7f15fc80c640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 0x7f15dc0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:45.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.660+0000 7f15fc80c640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 0x7f15dc0719e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f15f80c4530 tx=0x7f15f40079d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:45.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:45.676+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f15e805b3b0 con 0x7f15f80c9460 2024-04-25T11:41:46.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.736+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15d8005860 con 0x7f15f80c9460 2024-04-25T11:41:46.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.744+0000 7f15edffb640 1 -- 172.21.15.136:0/339176433 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7f15e805acb0 con 0x7f15f80c9460 2024-04-25T11:41:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:46.731 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:41:46.731 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:41:46.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.752+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 msgr2=0x7f15dc0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:46.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.752+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 0x7f15dc0719e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f15f80c4530 tx=0x7f15f40079d0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.752+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 msgr2=0x7f15f815f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:46.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.752+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f15e800b420 tx=0x7f15e800b8e0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 shutdown_connections 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15f81561e0 0x7f15f81ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f15dc06f4d0 0x7f15dc0719e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15f80ca110 0x7f15f80c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 --2- 172.21.15.136:0/339176433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15f80c9460 0x7f15f815f680 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 >> 172.21.15.136:0/339176433 conn(0x7f15f80d1290 msgr2=0x7f15f80cfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 shutdown_connections 2024-04-25T11:41:46.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:46.756+0000 7f15ff271640 1 -- 172.21.15.136:0/339176433 wait complete. 2024-04-25T11:41:46.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:41:48.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:49.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.124+0000 7ff023fff640 1 Processor -- start 2024-04-25T11:41:49.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.128+0000 7ff023fff640 1 -- start start 2024-04-25T11:41:49.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff023fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff01c0334f0 0x7ff01c006400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff023fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c0e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff023fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01c0df820 con 0x7ff01c04ba50 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff023fff640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff01c04c510 con 0x7ff01c0334f0 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff023fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff01c0de930 con 0x7ff01c006940 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff01c0334f0 0x7ff01c006400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff01c0334f0 0x7ff01c006400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48840/0 (socket says 172.21.15.136:48840) 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 -- 172.21.15.136:0/1619304201 learned_addr learned my addr 172.21.15.136:0/1619304201 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 -- 172.21.15.136:0/1619304201 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 msgr2=0x7ff01c0e5cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 --2- 172.21.15.136:0/1619304201 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c0e5cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 -- 172.21.15.136:0/1619304201 >> v1:172.21.15.157:6789/0 conn(0x7ff01c04ba50 legacy=0x7ff01c0df110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:49.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.132+0000 7ff0227fc640 1 -- 172.21.15.136:0/1619304201 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c0351a0 con 0x7ff01c0334f0 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff0227fc640 1 --2- 172.21.15.136:0/1619304201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c0334f0 0x7ff01c006400 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6c0889eb9c866d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff021ffb640 1 -- 172.21.15.136:0/1619304201 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140b0070 con 0x7ff01c0334f0 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff021ffb640 1 -- 172.21.15.136:0/1619304201 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0140ab600 con 0x7ff01c0334f0 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff021ffb640 1 -- 172.21.15.136:0/1619304201 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140ab9d0 con 0x7ff01c0334f0 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff023fff640 1 -- 172.21.15.136:0/1619304201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c0334f0 msgr2=0x7ff01c006400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.136+0000 7ff023fff640 1 --2- 172.21.15.136:0/1619304201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c0334f0 0x7ff01c006400 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01401c540 comp rx=0 tx=0).stop 2024-04-25T11:41:49.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.144+0000 7ff023fff640 1 -- 172.21.15.136:0/1619304201 shutdown_connections 2024-04-25T11:41:49.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/1619304201 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c0e5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:49.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/1619304201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c0334f0 0x7ff01c006400 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:49.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.144+0000 7ff023fff640 1 -- 172.21.15.136:0/1619304201 >> 172.21.15.136:0/1619304201 conn(0x7ff01c0503c0 msgr2=0x7ff01c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:49.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.152+0000 7ff023fff640 1 -- 172.21.15.136:0/1619304201 shutdown_connections 2024-04-25T11:41:49.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.152+0000 7ff023fff640 1 -- 172.21.15.136:0/1619304201 wait complete. 2024-04-25T11:41:49.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.152+0000 7ff023fff640 1 Processor -- start 2024-04-25T11:41:49.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.156+0000 7ff023fff640 1 -- start start 2024-04-25T11:41:49.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff022ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff022ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58768/0 (socket says 172.21.15.136:58768) 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0227fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0227fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37138/0 (socket says 172.21.15.136:37138) 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:48856/0 (socket says 172.21.15.136:48856) 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff022ffd640 1 -- 172.21.15.136:0/465396876 learned_addr learned my addr 172.21.15.136:0/465396876 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c04ad40 con 0x7ff01c04ba50 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c034ea0 con 0x7ff01c0334f0 2024-04-25T11:41:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff01c04c800 con 0x7ff01c006940 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 -- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 msgr2=0x7ff01c17ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 -- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 msgr2=0x7ff01c0d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c17ef90 con 0x7ff01c04ba50 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff0237fe640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff0240520f0 tx=0x7ff02406aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.160+0000 7ff022ffd640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.164+0000 7ff0227fc640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:49.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.176+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024091070 con 0x7ff01c04ba50 2024-04-25T11:41:49.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.176+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02406ddd0 con 0x7ff01c04ba50 2024-04-25T11:41:49.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.176+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0240955d0 con 0x7ff01c04ba50 2024-04-25T11:41:49.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.176+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c180520 con 0x7ff01c04ba50 2024-04-25T11:41:49.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.184+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c180d80 con 0x7ff01c04ba50 2024-04-25T11:41:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.184+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff024094070 con 0x7ff01c04ba50 2024-04-25T11:41:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.184+0000 7ff0137fe640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 0x7ff00c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.184+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7ff02410b480 con 0x7ff01c04ba50 2024-04-25T11:41:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.184+0000 7ff022ffd640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 0x7ff00c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.188+0000 7ff022ffd640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 0x7ff00c071ad0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff018006fd0 tx=0x7ff018008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:49.187 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.204+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c180830 con 0x7ff01c04ba50 2024-04-25T11:41:49.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:49.224+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff0240d79c0 con 0x7ff01c04ba50 2024-04-25T11:41:50.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.140+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff01c005580 con 0x7ff01c04ba50 2024-04-25T11:41:50.127 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.140+0000 7ff0137fe640 1 -- 172.21.15.136:0/465396876 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7ff0240d72c0 con 0x7ff01c04ba50 2024-04-25T11:41:50.129 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:50.129 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:41:50.129 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:41:50.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 msgr2=0x7ff00c071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:50.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 0x7ff00c071ad0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff018006fd0 tx=0x7ff018008040 comp rx=0 tx=0).stop 2024-04-25T11:41:50.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 msgr2=0x7ff01c17f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff0240520f0 tx=0x7ff02406aca0 comp rx=0 tx=0).stop 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 shutdown_connections 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff01c04ba50 0x7ff01c17f820 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff01c0334f0 0x7ff01c0d8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.144+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff00c06f5c0 0x7ff00c071ad0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.148+0000 7ff023fff640 1 --2- 172.21.15.136:0/465396876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff01c006940 0x7ff01c17ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:50.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.148+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 >> 172.21.15.136:0/465396876 conn(0x7ff01c0503c0 msgr2=0x7ff01c0dd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:50.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.156+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 shutdown_connections 2024-04-25T11:41:50.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:50.156+0000 7ff023fff640 1 -- 172.21.15.136:0/465396876 wait complete. 2024-04-25T11:41:50.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:41:52.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:52.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.604+0000 7fbd26535640 1 Processor -- start 2024-04-25T11:41:52.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.612+0000 7fbd26535640 1 -- start start 2024-04-25T11:41:52.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd26535640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbd2015c7a0 0x7fbd2015cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd26535640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd26535640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd200c3e40 con 0x7fbd20152db0 2024-04-25T11:41:52.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd26535640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2015d0f0 con 0x7fbd2015c7a0 2024-04-25T11:41:52.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd26535640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbd200c4100 con 0x7fbd201614a0 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58782/0 (socket says 172.21.15.136:58782) 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 -- 172.21.15.136:0/519964495 learned_addr learned my addr 172.21.15.136:0/519964495 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 -- 172.21.15.136:0/519964495 >> v2:172.21.15.157:3300/0 conn(0x7fbd2015c7a0 msgr2=0x7fbd2015cbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 --2- 172.21.15.136:0/519964495 >> v2:172.21.15.157:3300/0 conn(0x7fbd2015c7a0 0x7fbd2015cbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 -- 172.21.15.136:0/519964495 >> v1:172.21.15.136:6789/0 conn(0x7fbd20152db0 legacy=0x7fbd200c99b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.616+0000 7fbd25d34640 1 -- 172.21.15.136:0/519964495 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd20072b60 con 0x7fbd201614a0 2024-04-25T11:41:52.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd25d34640 1 --2- 172.21.15.136:0/519964495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009870 tx=0x7fbd1c01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c10d3f2d152ba085 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:52.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd17fff640 1 -- 172.21.15.136:0/519964495 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0b0070 con 0x7fbd201614a0 2024-04-25T11:41:52.605 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd17fff640 1 -- 172.21.15.136:0/519964495 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0ab600 con 0x7fbd201614a0 2024-04-25T11:41:52.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 -- 172.21.15.136:0/519964495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 msgr2=0x7fbd201638c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:52.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd17fff640 1 -- 172.21.15.136:0/519964495 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0b0070 con 0x7fbd201614a0 2024-04-25T11:41:52.606 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 --2- 172.21.15.136:0/519964495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009870 tx=0x7fbd1c01c540 comp rx=0 tx=0).stop 2024-04-25T11:41:52.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 -- 172.21.15.136:0/519964495 shutdown_connections 2024-04-25T11:41:52.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 --2- 172.21.15.136:0/519964495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd201614a0 0x7fbd201638c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:52.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 --2- 172.21.15.136:0/519964495 >> v2:172.21.15.157:3300/0 conn(0x7fbd2015c7a0 0x7fbd2015cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:52.607 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.620+0000 7fbd26535640 1 -- 172.21.15.136:0/519964495 >> 172.21.15.136:0/519964495 conn(0x7fbd2006d100 msgr2=0x7fbd20070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:52.609 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.624+0000 7fbd26535640 1 -- 172.21.15.136:0/519964495 shutdown_connections 2024-04-25T11:41:52.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.624+0000 7fbd26535640 1 -- 172.21.15.136:0/519964495 wait complete. 2024-04-25T11:41:52.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.628+0000 7fbd26535640 1 Processor -- start 2024-04-25T11:41:52.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 -- start start 2024-04-25T11:41:52.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd2015c7a0 0x7fbd201403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 0x7fbd20140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbd200c9410 con 0x7fbd201614a0 2024-04-25T11:41:52.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbd200c2850 con 0x7fbd2015c7a0 2024-04-25T11:41:52.618 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd26535640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbd200c1f40 con 0x7fbd20152db0 2024-04-25T11:41:52.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd25533640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:52.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd25d34640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 0x7fbd20140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:52.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd25533640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:58790/0 (socket says 172.21.15.136:58790) 2024-04-25T11:41:52.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd25533640 1 -- 172.21.15.136:0/2043510125 learned_addr learned my addr 172.21.15.136:0/2043510125 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:52.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.632+0000 7fbd24d32640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd2015c7a0 0x7fbd201403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 -- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd2015c7a0 msgr2=0x7fbd201403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd2015c7a0 0x7fbd201403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 -- 172.21.15.136:0/2043510125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 msgr2=0x7fbd20140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 0x7fbd20140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 -- 172.21.15.136:0/2043510125 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd20140e70 con 0x7fbd20152db0 2024-04-25T11:41:52.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25533640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800ba50 tx=0x7fbd1800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:52.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.636+0000 7fbd25d34640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 0x7fbd20140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:41:52.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.648+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18019070 con 0x7fbd20152db0 2024-04-25T11:41:52.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.652+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1800cbb0 con 0x7fbd20152db0 2024-04-25T11:41:52.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.652+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18014440 con 0x7fbd20152db0 2024-04-25T11:41:52.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.652+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd20161c30 con 0x7fbd20152db0 2024-04-25T11:41:52.636 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.652+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd20150590 con 0x7fbd20152db0 2024-04-25T11:41:52.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.668+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd000052a0 con 0x7fbd20152db0 2024-04-25T11:41:52.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.668+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbd1801c070 con 0x7fbd20152db0 2024-04-25T11:41:52.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.668+0000 7fbd15ffb640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 0x7fbd04071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:52.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.672+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7fbd18057e50 con 0x7fbd20152db0 2024-04-25T11:41:52.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.672+0000 7fbd24d32640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 0x7fbd04071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:52.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.676+0000 7fbd24d32640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 0x7fbd04071960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbd20163670 tx=0x7fbd10009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:52.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:52.676+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbd1805b860 con 0x7fbd20152db0 2024-04-25T11:41:53.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.668+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd00005860 con 0x7fbd20152db0 2024-04-25T11:41:53.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.668+0000 7fbd15ffb640 1 -- 172.21.15.136:0/2043510125 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7fbd1805fb60 con 0x7fbd20152db0 2024-04-25T11:41:53.653 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:53.653 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:41:53.653 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:41:53.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.676+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 msgr2=0x7fbd04071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 0x7fbd04071960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbd20163670 tx=0x7fbd10009060 comp rx=0 tx=0).stop 2024-04-25T11:41:53.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 msgr2=0x7fbd2015b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:53.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800ba50 tx=0x7fbd1800bf10 comp rx=0 tx=0).stop 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 shutdown_connections 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd201614a0 0x7fbd20140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbd2015c7a0 0x7fbd201403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbd0406f450 0x7fbd04071960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 --2- 172.21.15.136:0/2043510125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbd20152db0 0x7fbd2015b6c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.684+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 >> 172.21.15.136:0/2043510125 conn(0x7fbd2006d100 msgr2=0x7fbd200c9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.688+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 shutdown_connections 2024-04-25T11:41:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:53.688+0000 7fbd26535640 1 -- 172.21.15.136:0/2043510125 wait complete. 2024-04-25T11:41:53.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:41:55.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:56.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.300+0000 7fd1bccb8640 1 Processor -- start 2024-04-25T11:41:56.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.300+0000 7fd1bccb8640 1 -- start start 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd1b815a520 0x7fd1b8164550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd1b8072d80 0x7fd1b8160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b80ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b80c9990 con 0x7fd1b815a520 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b80c8ca0 con 0x7fd1b8072d80 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1bccb8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b81612c0 con 0x7fd1b80c9d70 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd1b815a520 0x7fd1b8164550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:56.290 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd1b815a520 0x7fd1b8164550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47812/0 (socket says 172.21.15.136:47812) 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 -- 172.21.15.136:0/993078945 learned_addr learned my addr 172.21.15.136:0/993078945 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 -- 172.21.15.136:0/993078945 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 msgr2=0x7fd1b80ca180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/993078945 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b80ca180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 -- 172.21.15.136:0/993078945 >> v2:172.21.15.157:3300/0 conn(0x7fd1b8072d80 msgr2=0x7fd1b8160d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/993078945 >> v2:172.21.15.157:3300/0 conn(0x7fd1b8072d80 0x7fd1b8160d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.304+0000 7fd1b77fe640 1 -- 172.21.15.136:0/993078945 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b80c86e0 con 0x7fd1b815a520 2024-04-25T11:41:56.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/993078945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b8164550 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8072b20 tx=0x7fd1b001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa51b3ce2aa47818 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:56.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1b67fc640 1 -- 172.21.15.136:0/993078945 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00a5620 con 0x7fd1b815a520 2024-04-25T11:41:56.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1b67fc640 1 -- 172.21.15.136:0/993078945 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00a57c0 con 0x7fd1b815a520 2024-04-25T11:41:56.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1b67fc640 1 -- 172.21.15.136:0/993078945 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00a5b90 con 0x7fd1b815a520 2024-04-25T11:41:56.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 -- 172.21.15.136:0/993078945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 msgr2=0x7fd1b8164550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:56.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 --2- 172.21.15.136:0/993078945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b8164550 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8072b20 tx=0x7fd1b001c9a0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 -- 172.21.15.136:0/993078945 shutdown_connections 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 --2- 172.21.15.136:0/993078945 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b80ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 --2- 172.21.15.136:0/993078945 >> v2:172.21.15.157:3300/0 conn(0x7fd1b8072d80 0x7fd1b8160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 --2- 172.21.15.136:0/993078945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b8164550 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 -- 172.21.15.136:0/993078945 >> 172.21.15.136:0/993078945 conn(0x7fd1b806c840 msgr2=0x7fd1b806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:56.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.316+0000 7fd1bccb8640 1 -- 172.21.15.136:0/993078945 shutdown_connections 2024-04-25T11:41:56.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.328+0000 7fd1bccb8640 1 -- 172.21.15.136:0/993078945 wait complete. 2024-04-25T11:41:56.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.328+0000 7fd1bccb8640 1 Processor -- start 2024-04-25T11:41:56.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.332+0000 7fd1bccb8640 1 -- start start 2024-04-25T11:41:56.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.336+0000 7fd1bccb8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.336+0000 7fd1bccb8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1bccb8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b80cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:40014/0 (socket says 172.21.15.136:40014) 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:56.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:51302/0 (socket says 172.21.15.136:51302) 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1bccb8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b80c82e0 con 0x7fd1b815a520 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1bccb8640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8073190 con 0x7fd1b8072d80 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 -- 172.21.15.136:0/1386792242 learned_addr learned my addr 172.21.15.136:0/1386792242 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b7fff640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b80cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1bccb8640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b80c8460 con 0x7fd1b80c9d70 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 -- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 msgr2=0x7fd1b815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 -- 172.21.15.136:0/1386792242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 msgr2=0x7fd1b80cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b80cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b8155a60 con 0x7fd1b8072d80 2024-04-25T11:41:56.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b6ffd640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:41:56.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.340+0000 7fd1b77fe640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001d240 tx=0x7fd1b0019740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:56.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.352+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00bc020 con 0x7fd1b8072d80 2024-04-25T11:41:56.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.352+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00b8670 con 0x7fd1b8072d80 2024-04-25T11:41:56.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.352+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00b8a20 con 0x7fd1b8072d80 2024-04-25T11:41:56.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.352+0000 7fd1bccb8640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b8151600 con 0x7fd1b8072d80 2024-04-25T11:41:56.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.352+0000 7fd1bccb8640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b8205c50 con 0x7fd1b8072d80 2024-04-25T11:41:56.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.360+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1900052a0 con 0x7fd1b8072d80 2024-04-25T11:41:56.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd1b00bf020 con 0x7fd1b8072d80 2024-04-25T11:41:56.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd19ffff640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 0x7fd1a40719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:56.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7681+0+0 (secure 0 0 0) 0x7fd1b012afd0 con 0x7fd1b8072d80 2024-04-25T11:41:56.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd1b00c6030 con 0x7fd1b8072d80 2024-04-25T11:41:56.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd1b6ffd640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 0x7fd1a40719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:56.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.364+0000 7fd1b6ffd640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 0x7fd1a40719e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81fcdd0 tx=0x7fd1ac006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:56.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:41:56.629 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:41:56.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.768+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd190005590 con 0x7fd1b8072d80 2024-04-25T11:41:56.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.772+0000 7fd19ffff640 1 -- 172.21.15.136:0/1386792242 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+6934 (secure 0 0 0) 0x7fd1b0016620 con 0x7fd1b8072d80 2024-04-25T11:41:56.758 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:56.758 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5911,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:39:46.402029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5408,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5408":{"gid":5408,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":21,"addr":"172.21.15.157:6837/1501623433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":1501623433},{"type":"v1","addr":"172.21.15.157:6837","nonce":1501623433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5408,"qdb_cluster":[6246,5517,5839,5449,5408]},"id":1}]} 2024-04-25T11:41:56.758 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 85 2024-04-25T11:41:56.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 msgr2=0x7fd1a40719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:56.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 0x7fd1a40719e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd1b81fcdd0 tx=0x7fd1ac006f90 comp rx=0 tx=0).stop 2024-04-25T11:41:56.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 msgr2=0x7fd1b8206450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:56.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001d240 tx=0x7fd1b0019740 comp rx=0 tx=0).stop 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 shutdown_connections 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1b815a520 0x7fd1b80cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1a406f4d0 0x7fd1a40719e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1b80c9d70 0x7fd1b815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 --2- 172.21.15.136:0/1386792242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1b8072d80 0x7fd1b8206450 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.784+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 >> 172.21.15.136:0/1386792242 conn(0x7fd1b806c840 msgr2=0x7fd1b8162260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.792+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 shutdown_connections 2024-04-25T11:41:56.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:56.800+0000 7fd19dffb640 1 -- 172.21.15.136:0/1386792242 wait complete. 2024-04-25T11:41:56.822 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T11:41:58.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:41:59.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.152+0000 7f0a0dad7640 1 Processor -- start 2024-04-25T11:41:59.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.152+0000 7f0a0dad7640 1 -- start start 2024-04-25T11:41:59.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0dad7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0a0004ba50 0x7f0a000df110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0dad7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a000334f0 0x7f0a00006400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0dad7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0004c510 con 0x7f0a00006940 2024-04-25T11:41:59.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0dad7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0a000de930 con 0x7f0a0004ba50 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0dad7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a000351a0 con 0x7f0a000334f0 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0a0004ba50 0x7f0a000df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0a0004ba50 0x7f0a000df110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47844/0 (socket says 172.21.15.136:47844) 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 -- 172.21.15.136:0/2684271749 learned_addr learned my addr 172.21.15.136:0/2684271749 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 -- 172.21.15.136:0/2684271749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a000334f0 msgr2=0x7f0a00006400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 --2- 172.21.15.136:0/2684271749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a000334f0 0x7f0a00006400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 -- 172.21.15.136:0/2684271749 >> v1:172.21.15.136:6789/0 conn(0x7f0a00006940 legacy=0x7f0a000db3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:41:59.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.156+0000 7f0a0cad5640 1 -- 172.21.15.136:0/2684271749 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a00035480 con 0x7f0a0004ba50 2024-04-25T11:41:59.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.160+0000 7f0a0cad5640 1 --2- 172.21.15.136:0/2684271749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a0004ba50 0x7f0a000df110 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000dfb0 tx=0x7f09fc025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b46285ae8eeeafe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:59.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.160+0000 7f0a077fe640 1 -- 172.21.15.136:0/2684271749 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0b3070 con 0x7f0a0004ba50 2024-04-25T11:41:59.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.160+0000 7f0a077fe640 1 -- 172.21.15.136:0/2684271749 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09fc0adda0 con 0x7f0a0004ba50 2024-04-25T11:41:59.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.164+0000 7f0a0dad7640 1 -- 172.21.15.136:0/2684271749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a0004ba50 msgr2=0x7f0a000df110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:59.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.164+0000 7f0a077fe640 1 -- 172.21.15.136:0/2684271749 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0b3070 con 0x7f0a0004ba50 2024-04-25T11:41:59.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.164+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/2684271749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a0004ba50 0x7f0a000df110 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000dfb0 tx=0x7f09fc025040 comp rx=0 tx=0).stop 2024-04-25T11:41:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 -- 172.21.15.136:0/2684271749 shutdown_connections 2024-04-25T11:41:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/2684271749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a000334f0 0x7f0a00006400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/2684271749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a0004ba50 0x7f0a000df110 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 -- 172.21.15.136:0/2684271749 >> 172.21.15.136:0/2684271749 conn(0x7f0a000503c0 msgr2=0x7f0a000507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:59.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 -- 172.21.15.136:0/2684271749 shutdown_connections 2024-04-25T11:41:59.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.168+0000 7f0a0dad7640 1 -- 172.21.15.136:0/2684271749 wait complete. 2024-04-25T11:41:59.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.180+0000 7f0a0dad7640 1 Processor -- start 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.180+0000 7f0a0dad7640 1 -- start start 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a00006940 0x7f0a001756f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 0x7f0a000d4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0004c800 con 0x7f0a000334f0 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0004ad40 con 0x7f0a00006940 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0dad7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0004c980 con 0x7f0a0004ba50 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a07fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 0x7f0a000d4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:59.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:51320/0 (socket says 172.21.15.136:51320) 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 -- 172.21.15.136:0/3066801067 learned_addr learned my addr 172.21.15.136:0/3066801067 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 -- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a00006940 msgr2=0x7f0a001756f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a00006940 0x7f0a001756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 -- 172.21.15.136:0/3066801067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 msgr2=0x7f0a000d4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 0x7f0a000d4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a0d2d6640 1 -- 172.21.15.136:0/3066801067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a000d5d30 con 0x7f0a0004ba50 2024-04-25T11:41:59.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.184+0000 7f0a07fff640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 0x7f0a000d4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:41:59.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.188+0000 7f0a0d2d6640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a0806c040 tx=0x7f0a0806aca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:59.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.192+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08091070 con 0x7f0a0004ba50 2024-04-25T11:41:59.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.192+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a00175c30 con 0x7f0a0004ba50 2024-04-25T11:41:59.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.192+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a000d8b80 con 0x7f0a0004ba50 2024-04-25T11:41:59.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.192+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0806dbd0 con 0x7f0a0004ba50 2024-04-25T11:41:59.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.192+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080955b0 con 0x7f0a0004ba50 2024-04-25T11:41:59.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.212+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0a08094070 con 0x7f0a0004ba50 2024-04-25T11:41:59.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.212+0000 7f0a057fa640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 0x7f09ec0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:41:59.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.212+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f0a0810be30 con 0x7f0a0004ba50 2024-04-25T11:41:59.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.212+0000 7f0a0cad5640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 0x7f09ec0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:41:59.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.220+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e80052a0 con 0x7f0a0004ba50 2024-04-25T11:41:59.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.220+0000 7f0a0cad5640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 0x7f09ec0719e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0b8000 tx=0x7f09fc004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:41:59.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.224+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0a080d84f0 con 0x7f0a0004ba50 2024-04-25T11:41:59.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.764+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e8005590 con 0x7f0a0004ba50 2024-04-25T11:41:59.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.768+0000 7f0a057fa640 1 -- 172.21.15.136:0/3066801067 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6101 (secure 0 0 0) 0x7f0a080d7df0 con 0x7f0a0004ba50 2024-04-25T11:41:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:41:59.753 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:41:59.046687+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:41:59.753 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 86 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 msgr2=0x7f09ec0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 0x7f09ec0719e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0b8000 tx=0x7f09fc004740 comp rx=0 tx=0).stop 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 msgr2=0x7f0a000cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0a0806c040 tx=0x7f0a0806aca0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 shutdown_connections 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f09ec06f4d0 0x7f09ec0719e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a0004ba50 0x7f0a000cc7d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a000334f0 0x7f0a000d4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 --2- 172.21.15.136:0/3066801067 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a00006940 0x7f0a001756f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:41:59.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 >> 172.21.15.136:0/3066801067 conn(0x7f0a000503c0 msgr2=0x7f0a00005940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:41:59.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 shutdown_connections 2024-04-25T11:41:59.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:41:59.772+0000 7f0a0dad7640 1 -- 172.21.15.136:0/3066801067 wait complete. 2024-04-25T11:41:59.828 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-25T11:41:59.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:41:59.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:00.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 Processor -- start 2024-04-25T11:42:00.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 -- start start 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4000160db0 0x7f40000c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f40000c9190 0x7f4000163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f400015a890 con 0x7f40000c9190 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f400006cbb0 con 0x7f4000160db0 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffffff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4000073860 con 0x7f400015a480 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:51334/0 (socket says 172.21.15.136:51334) 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 -- 172.21.15.136:0/2884728683 learned_addr learned my addr 172.21.15.136:0/2884728683 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:00.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 -- 172.21.15.136:0/2884728683 >> v2:172.21.15.157:3300/0 conn(0x7f4000160db0 msgr2=0x7f40000c8c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:00.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 --2- 172.21.15.136:0/2884728683 >> v2:172.21.15.157:3300/0 conn(0x7f4000160db0 0x7f40000c8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 -- 172.21.15.136:0/2884728683 >> v2:172.21.15.136:3300/0 conn(0x7f40000c9190 msgr2=0x7f4000163950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:00.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 --2- 172.21.15.136:0/2884728683 >> v2:172.21.15.136:3300/0 conn(0x7f40000c9190 0x7f4000163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.123 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.136+0000 7f3ffeffd640 1 -- 172.21.15.136:0/2884728683 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40000c5cc0 con 0x7f400015a480 2024-04-25T11:42:00.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.140+0000 7f3ffeffd640 1 --2- 172.21.15.136:0/2884728683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40000ceb20 tx=0x7f3ff001ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88620258138a0ce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:00.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.140+0000 7f3ffdffb640 1 -- 172.21.15.136:0/2884728683 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00b5070 con 0x7f400015a480 2024-04-25T11:42:00.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.140+0000 7f3ffdffb640 1 -- 172.21.15.136:0/2884728683 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff00a89f0 con 0x7f400015a480 2024-04-25T11:42:00.125 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.140+0000 7f3ffdffb640 1 -- 172.21.15.136:0/2884728683 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00a8dc0 con 0x7f400015a480 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 -- 172.21.15.136:0/2884728683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 msgr2=0x7f400006c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 --2- 172.21.15.136:0/2884728683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40000ceb20 tx=0x7f3ff001ed50 comp rx=0 tx=0).stop 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 -- 172.21.15.136:0/2884728683 shutdown_connections 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 --2- 172.21.15.136:0/2884728683 >> v2:172.21.15.136:3300/0 conn(0x7f40000c9190 0x7f4000163950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 --2- 172.21.15.136:0/2884728683 >> v2:172.21.15.157:3300/0 conn(0x7f4000160db0 0x7f40000c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 --2- 172.21.15.136:0/2884728683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f400015a480 0x7f400006c670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.130 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.144+0000 7f3ffffff640 1 -- 172.21.15.136:0/2884728683 >> 172.21.15.136:0/2884728683 conn(0x7f40000cdb40 msgr2=0x7f40000cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:00.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 -- 172.21.15.136:0/2884728683 shutdown_connections 2024-04-25T11:42:00.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 -- 172.21.15.136:0/2884728683 wait complete. 2024-04-25T11:42:00.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 Processor -- start 2024-04-25T11:42:00.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 -- start start 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 0x7f4000152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 0x7f40001f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 0x7f4000152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 0x7f4000152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47872/0 (socket says 172.21.15.136:47872) 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffeffd640 1 -- 172.21.15.136:0/3070551445 learned_addr learned my addr 172.21.15.136:0/3070551445 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.148+0000 7f3ffffff640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4000160db0 0x7f400015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffffff640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4000073ab0 con 0x7f40000c9190 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffffff640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40000743e0 con 0x7f400015a480 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffffff640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4000073d90 con 0x7f4000160db0 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 0x7f40001f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:00.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fff7fe640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4000160db0 0x7f400015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 -- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4000160db0 msgr2=0x7f400015b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4000160db0 0x7f400015b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 -- 172.21.15.136:0/3070551445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 msgr2=0x7f4000152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 0x7f4000152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40001f7250 con 0x7f400015a480 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffe7fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 0x7f40001f6d10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b440 tx=0x7f3ff400b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4015020 con 0x7f400015a480 2024-04-25T11:42:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4011850 con 0x7f400015a480 2024-04-25T11:42:00.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4011c40 con 0x7f400015a480 2024-04-25T11:42:00.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffffff640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40001f9f40 con 0x7f400015a480 2024-04-25T11:42:00.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3ffffff640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40001fa4e0 con 0x7f400015a480 2024-04-25T11:42:00.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd80052a0 con 0x7f400015a480 2024-04-25T11:42:00.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3ff4018020 con 0x7f400015a480 2024-04-25T11:42:00.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 0x7f3fec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:00.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.152+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f3ff408f8a0 con 0x7f400015a480 2024-04-25T11:42:00.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.156+0000 7f3ffeffd640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 0x7f3fec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:00.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.156+0000 7f3ffeffd640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 0x7f3fec071940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00097f0 tx=0x7f3ff00aa070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:00.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.160+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3ff401f020 con 0x7f400015a480 2024-04-25T11:42:00.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.788+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fd8005590 con 0x7f400015a480 2024-04-25T11:42:00.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.788+0000 7f3fe7fff640 1 -- 172.21.15.136:0/3070551445 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6101 (secure 0 0 0) 0x7f3ff405b920 con 0x7f400015a480 2024-04-25T11:42:00.779 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:00.779 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:41:59.046687+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:42:00.782 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 86 2024-04-25T11:42:00.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 msgr2=0x7f3fec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:00.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 0x7f3fec071940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00097f0 tx=0x7f3ff00aa070 comp rx=0 tx=0).stop 2024-04-25T11:42:00.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 msgr2=0x7f40001f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:00.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 0x7f40001f6d10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400b440 tx=0x7f3ff400b900 comp rx=0 tx=0).stop 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 shutdown_connections 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3fec06f430 0x7f3fec071940 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4000160db0 0x7f400015b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f400015a480 0x7f40001f6d10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 --2- 172.21.15.136:0/3070551445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40000c9190 0x7f4000152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 >> 172.21.15.136:0/3070551445 conn(0x7f40000cdb40 msgr2=0x7f400015f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 shutdown_connections 2024-04-25T11:42:00.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:00.804+0000 7f3fe67fc640 1 -- 172.21.15.136:0/3070551445 wait complete. 2024-04-25T11:42:00.830 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:41:59.046687+0000", "qdb_cluster": [ 5449, 5839, 5517, 6246 ], "qdb_leader": 5449, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5449, "mds_2": 5517, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:42:02.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:03.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.260+0000 7f9f8e6ab640 1 Processor -- start 2024-04-25T11:42:03.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.260+0000 7f9f8e6ab640 1 -- start start 2024-04-25T11:42:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8e6ab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9f88152e10 0x7f9f880cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8e6ab640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f88153220 0x7f9f88073c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8e6ab640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f880cdb50 con 0x7f9f880741c0 2024-04-25T11:42:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8e6ab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9f880c60a0 con 0x7f9f88152e10 2024-04-25T11:42:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8e6ab640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f880ccdd0 con 0x7f9f88153220 2024-04-25T11:42:03.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9f880741c0 0x7f9f88158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:33842/0 (socket says 172.21.15.136:33842) 2024-04-25T11:42:03.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 -- 172.21.15.136:0/982484860 learned_addr learned my addr 172.21.15.136:0/982484860 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:03.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f87fff640 1 --2- 172.21.15.136:0/982484860 >> v2:172.21.15.136:3300/0 conn(0x7f9f88152e10 0x7f9f880cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343015936 0 0) 0x7f9f880cdb50 con 0x7f9f880741c0 2024-04-25T11:42:03.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70003530 con 0x7f9f880741c0 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1307525704 0 0) 0x7f9f70003530 con 0x7f9f880741c0 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f88153220 msgr2=0x7f9f88073c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f88153220 0x7f9f88073c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 >> v2:172.21.15.136:3300/0 conn(0x7f9f88152e10 msgr2=0x7f9f880cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 --2- 172.21.15.136:0/982484860 >> v2:172.21.15.136:3300/0 conn(0x7f9f88152e10 0x7f9f880cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f880cc6e0 con 0x7f9f880741c0 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f9f80001a00 con 0x7f9f880741c0 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 >> v1:172.21.15.157:6789/0 conn(0x7f9f880741c0 legacy=0x7f9f88158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f700039c0 0x7f9f70003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f70004810 0x7f9f70004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f7000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f880c60a0 con 0x7f9f70005160 2024-04-25T11:42:03.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f880ccdd0 con 0x7f9f700039c0 2024-04-25T11:42:03.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7000c100 con 0x7f9f70004810 2024-04-25T11:42:03.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f87fff640 1 --2- 172.21.15.136:0/982484860 >> v2:172.21.15.136:3300/0 conn(0x7f9f88152e10 0x7f9f880cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:03.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f7000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 -- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f70004810 msgr2=0x7f9f70004c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:03.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f70004810 0x7f9f70004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 -- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f700039c0 msgr2=0x7f9f70003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:03.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f700039c0 0x7f9f70003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.264+0000 7f9f8cc48640 1 -- 172.21.15.136:0/982484860 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f880cc6e0 con 0x7f9f70005160 2024-04-25T11:42:03.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.268+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f7000b9d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9f880c8970 tx=0x7f9f8001d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7a0c083b8506c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:03.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.268+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8000a570 con 0x7f9f70005160 2024-04-25T11:42:03.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.268+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f800a9020 con 0x7f9f70005160 2024-04-25T11:42:03.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.268+0000 7f9f86ffd640 1 -- 172.21.15.136:0/982484860 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f800072f0 con 0x7f9f70005160 2024-04-25T11:42:03.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/982484860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 msgr2=0x7f9f7000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/982484860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f7000b9d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9f880c8970 tx=0x7f9f8001d3c0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8cc48640 1 -- 172.21.15.136:0/982484860 reap_dead start 2024-04-25T11:42:03.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/982484860 shutdown_connections 2024-04-25T11:42:03.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/982484860 >> 172.21.15.136:0/982484860 conn(0x7f9f88068760 msgr2=0x7f9f8806d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:03.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.272+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/982484860 shutdown_connections 2024-04-25T11:42:03.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.276+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/982484860 wait complete. 2024-04-25T11:42:03.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.276+0000 7f9f8e6ab640 1 Processor -- start 2024-04-25T11:42:03.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.280+0000 7f9f8e6ab640 1 -- start start 2024-04-25T11:42:03.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f880c2cf0 0x7f9f8814f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 0x7f9f8814f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f880cd630 con 0x7f9f70005160 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9f880cd350 con 0x7f9f880c2cf0 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8e6ab640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9f880cc9c0 con 0x7f9f8814c5d0 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47910/0 (socket says 172.21.15.136:47910) 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 -- 172.21.15.136:0/2001693513 learned_addr learned my addr 172.21.15.136:0/2001693513 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 0x7f9f8814f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f877fe640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f880c2cf0 0x7f9f8814f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 -- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 msgr2=0x7f9f8814f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 0x7f9f8814f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 -- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f880c2cf0 msgr2=0x7f9f8814f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f880c2cf0 0x7f9f8814f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f87fff640 1 -- 172.21.15.136:0/2001693513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f88159370 con 0x7f9f70005160 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.284+0000 7f9f8cc48640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 0x7f9f8814f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f87fff640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800ec00 tx=0x7f9f7800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78017070 con 0x7f9f70005160 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7800ee10 con 0x7f9f70005160 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f88159650 con 0x7f9f70005160 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7800ce10 con 0x7f9f70005160 2024-04-25T11:42:03.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f880c3780 con 0x7f9f70005160 2024-04-25T11:42:03.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8806a2e0 con 0x7f9f70005160 2024-04-25T11:42:03.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.288+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9f7801a070 con 0x7f9f70005160 2024-04-25T11:42:03.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.292+0000 7f9f84ff9640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 0x7f9f74071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:03.276 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.292+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f9f7808f9f0 con 0x7f9f70005160 2024-04-25T11:42:03.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.292+0000 7f9f877fe640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 0x7f9f74071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:03.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.292+0000 7f9f877fe640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 0x7f9f74071940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9f88075340 tx=0x7f9f7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:03.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.304+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f7805bab0 con 0x7f9f70005160 2024-04-25T11:42:03.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.780+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f880c0b90 con 0x7f9f70005160 2024-04-25T11:42:03.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.780+0000 7f9f84ff9640 1 -- 172.21.15.136:0/2001693513 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6101 (secure 0 0 0) 0x7f9f7805fdb0 con 0x7f9f70005160 2024-04-25T11:42:03.767 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:03.767 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:41:59.046687+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:42:03.767 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 86 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 msgr2=0x7f9f74071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 0x7f9f74071940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9f88075340 tx=0x7f9f7c008040 comp rx=0 tx=0).stop 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 msgr2=0x7f9f8814bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800ec00 tx=0x7f9f7800c520 comp rx=0 tx=0).stop 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 shutdown_connections 2024-04-25T11:42:03.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9f7406f430 0x7f9f74071940 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9f8814c5d0 0x7f9f8814f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f880c2cf0 0x7f9f8814f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 --2- 172.21.15.136:0/2001693513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f70005160 0x7f9f8814bf30 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 >> 172.21.15.136:0/2001693513 conn(0x7f9f88068760 msgr2=0x7f9f88071040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 shutdown_connections 2024-04-25T11:42:03.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:03.784+0000 7f9f8e6ab640 1 -- 172.21.15.136:0/2001693513 wait complete. 2024-04-25T11:42:03.806 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:05.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:06.124 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.136+0000 7f044f0c5640 1 Processor -- start 2024-04-25T11:42:06.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.144+0000 7f044f0c5640 1 -- start start 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 0x7f0448070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f044815b5d0 0x7f044815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f04480c8ee0 con 0x7f044815b5d0 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0448152700 con 0x7f04481522f0 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044f0c5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04480729c0 con 0x7f04480c9370 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56444/0 (socket says 172.21.15.136:56444) 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 -- 172.21.15.136:0/1644018842 learned_addr learned my addr 172.21.15.136:0/1644018842 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 -- 172.21.15.136:0/1644018842 >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 msgr2=0x7f0448070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f044ce61640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 0x7f0448070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 0x7f0448070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 -- 172.21.15.136:0/1644018842 >> v2:172.21.15.136:3300/0 conn(0x7f044815b5d0 msgr2=0x7f044815b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:06.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.136:3300/0 conn(0x7f044815b5d0 0x7f044815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.148+0000 7f0447fff640 1 -- 172.21.15.136:0/1644018842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0448073f30 con 0x7f04480c9370 2024-04-25T11:42:06.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.152+0000 7f044ce61640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 0x7f0448070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:06.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.164+0000 7f0447fff640 1 --2- 172.21.15.136:0/1644018842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0448071ee0 tx=0x7f043c020be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c54de74be8c416 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.164+0000 7f04477fe640 1 -- 172.21.15.136:0/1644018842 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c0af070 con 0x7f04480c9370 2024-04-25T11:42:06.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f04477fe640 1 -- 172.21.15.136:0/1644018842 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f043c0a9b60 con 0x7f04480c9370 2024-04-25T11:42:06.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 -- 172.21.15.136:0/1644018842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 msgr2=0x7f04480c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 --2- 172.21.15.136:0/1644018842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0448071ee0 tx=0x7f043c020be0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 -- 172.21.15.136:0/1644018842 shutdown_connections 2024-04-25T11:42:06.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.136:3300/0 conn(0x7f044815b5d0 0x7f044815b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 --2- 172.21.15.136:0/1644018842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04480c9370 0x7f04480c9780 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0448071ee0 tx=0x7f043c020be0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 --2- 172.21.15.136:0/1644018842 >> v2:172.21.15.157:3300/0 conn(0x7f04481522f0 0x7f0448070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.172+0000 7f044f0c5640 1 -- 172.21.15.136:0/1644018842 >> 172.21.15.136:0/1644018842 conn(0x7f04480c9d50 msgr2=0x7f04480ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.176+0000 7f044f0c5640 1 -- 172.21.15.136:0/1644018842 shutdown_connections 2024-04-25T11:42:06.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.176+0000 7f044f0c5640 1 -- 172.21.15.136:0/1644018842 wait complete. 2024-04-25T11:42:06.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 Processor -- start 2024-04-25T11:42:06.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 -- start start 2024-04-25T11:42:06.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 0x7f044814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0448151910 0x7f04481fa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0448158970 con 0x7f044815b5d0 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04481587f0 con 0x7f04481522f0 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044f0c5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0448158af0 con 0x7f0448151910 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:38904/0 (socket says 172.21.15.136:38904) 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 -- 172.21.15.136:0/629008238 learned_addr learned my addr 172.21.15.136:0/629008238 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f0447fff640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 0x7f044814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 -- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0448151910 msgr2=0x7f04481fa5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0448151910 0x7f04481fa5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 -- 172.21.15.136:0/629008238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 msgr2=0x7f044814f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 0x7f044814f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f044ce61640 1 -- 172.21.15.136:0/629008238 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04481fab10 con 0x7f04481522f0 2024-04-25T11:42:06.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.180+0000 7f0447fff640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 0x7f044814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:06.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f044ce61640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f043800c850 tx=0x7f043800cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438015070 con 0x7f04481522f0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0438007830 con 0x7f04481522f0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438007c40 con 0x7f04481522f0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04480c29f0 con 0x7f04481522f0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04480c2fb0 con 0x7f04481522f0 2024-04-25T11:42:06.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04481f9ea0 con 0x7f04481522f0 2024-04-25T11:42:06.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.184+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0438018070 con 0x7f04481522f0 2024-04-25T11:42:06.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.188+0000 7f04457fa640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 0x7f0434071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:06.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.188+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f043808fc70 con 0x7f04481522f0 2024-04-25T11:42:06.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.188+0000 7f0447fff640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 0x7f0434071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:06.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.188+0000 7f0447fff640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 0x7f0434071940 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f043c0b4000 tx=0x7f043c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:06.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.192+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f043805be70 con 0x7f04481522f0 2024-04-25T11:42:06.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.612+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f044806a350 con 0x7f04481522f0 2024-04-25T11:42:06.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.612+0000 7f04457fa640 1 -- 172.21.15.136:0/629008238 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6103 (secure 0 0 0) 0x7f0438060170 con 0x7f04481522f0 2024-04-25T11:42:06.598 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:06.598 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:05.650156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:06.599 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 87 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 msgr2=0x7f0434071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 0x7f0434071940 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f043c0b4000 tx=0x7f043c004740 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 msgr2=0x7f04481eb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f043800c850 tx=0x7f043800cd10 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 shutdown_connections 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f043406f430 0x7f0434071940 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0448151910 0x7f04481fa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f044815b5d0 0x7f044814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 --2- 172.21.15.136:0/629008238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04481522f0 0x7f04481eb2b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.616+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 >> 172.21.15.136:0/629008238 conn(0x7f04480c9d50 msgr2=0x7f04480c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.620+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 shutdown_connections 2024-04-25T11:42:06.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:06.620+0000 7f044f0c5640 1 -- 172.21.15.136:0/629008238 wait complete. 2024-04-25T11:42:06.641 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:08.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:08.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 Processor -- start 2024-04-25T11:42:08.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 -- start start 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc014073ac0 0x7fc014160c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0140c9340 con 0x7fc01406c670 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0140c95c0 con 0x7fc01415a600 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01c15e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc0140cae20 con 0x7fc014073ac0 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01a6fb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc01406c670 0x7fc01416aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47748/0 (socket says 172.21.15.136:47748) 2024-04-25T11:42:08.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.952+0000 7fc01a6fb640 1 -- 172.21.15.136:0/405700642 learned_addr learned my addr 172.21.15.136:0/405700642 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154395368 0 0) 0x7fc010001120 con 0x7fc01406c670 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc000003530 con 0x7fc01406c670 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3702143486 0 0) 0x7fc000003530 con 0x7fc01406c670 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 >> v2:172.21.15.136:3300/0 conn(0x7fc014073ac0 msgr2=0x7fc014160c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 --2- 172.21.15.136:0/405700642 >> v2:172.21.15.136:3300/0 conn(0x7fc014073ac0 0x7fc014160c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 >> v1:172.21.15.157:6789/0 conn(0x7fc01415a600 legacy=0x7fc0140cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0140c98a0 con 0x7fc01406c670 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fc010001850 con 0x7fc01406c670 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 >> v1:172.21.15.136:6789/0 conn(0x7fc01406c670 legacy=0x7fc01416aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc000003a10 0x7fc000003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc000004810 0x7fc000004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc00000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140cae20 con 0x7fc0000053b0 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc00001d340 con 0x7fc000004810 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc00001d540 con 0x7fc000003a10 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc00000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc0196f9640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc000004810 0x7fc000004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 -- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc000003a10 msgr2=0x7fc000003e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc000003a10 0x7fc000003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 -- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc000004810 msgr2=0x7fc000004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc000004810 0x7fc000004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.956+0000 7fc01a6fb640 1 -- 172.21.15.136:0/405700642 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc000003530 con 0x7fc0000053b0 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.960+0000 7fc01a6fb640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc00000ba60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc010007e00 tx=0x7fc01003ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99369e873d2586f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.960+0000 7fc01a6fb640 1 -- 172.21.15.136:0/405700642 reap_dead start 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.960+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010007e40 con 0x7fc0000053b0 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.960+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0100cb030 con 0x7fc0000053b0 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.960+0000 7fc018ef8640 1 -- 172.21.15.136:0/405700642 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0100058e0 con 0x7fc0000053b0 2024-04-25T11:42:08.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 -- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 msgr2=0x7fc00000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc00000ba60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc010007e00 tx=0x7fc01003ee30 comp rx=0 tx=0).stop 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 -- 172.21.15.136:0/405700642 shutdown_connections 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 --2- 172.21.15.136:0/405700642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc00000ba60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 -- 172.21.15.136:0/405700642 >> 172.21.15.136:0/405700642 conn(0x7fc01406d780 msgr2=0x7fc01406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 -- 172.21.15.136:0/405700642 shutdown_connections 2024-04-25T11:42:08.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.964+0000 7fc01c15e640 1 -- 172.21.15.136:0/405700642 wait complete. 2024-04-25T11:42:08.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.968+0000 7fc01c15e640 1 Processor -- start 2024-04-25T11:42:08.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.968+0000 7fc01c15e640 1 -- start start 2024-04-25T11:42:08.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc014149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 0x7fc014147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc014070610 con 0x7fc0000053b0 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140caab0 con 0x7fc01414d5d0 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01c15e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc014070790 con 0x7fc014156980 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01a6fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01a6fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56454/0 (socket says 172.21.15.136:56454) 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01a6fb640 1 -- 172.21.15.136:0/1990741174 learned_addr learned my addr 172.21.15.136:0/1990741174 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:08.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 0x7fc014147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 -- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 msgr2=0x7fc0141f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 -- 172.21.15.136:0/1990741174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 msgr2=0x7fc014149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc014149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 -- 172.21.15.136:0/1990741174 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc014155630 con 0x7fc01414d5d0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc01a6fb640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.972+0000 7fc0196f9640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 0x7fc014147370 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc00400f5c0 tx=0x7fc00400fa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc004017030 con 0x7fc01414d5d0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fc01c15e640 1 -- 172.21.15.136:0/1990741174 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01414ba30 con 0x7fc01414d5d0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc004003a80 con 0x7fc01414d5d0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc004003c50 con 0x7fc01414d5d0 2024-04-25T11:42:08.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fc01c15e640 1 -- 172.21.15.136:0/1990741174 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01414bff0 con 0x7fc01414d5d0 2024-04-25T11:42:08.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.976+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff00052a0 con 0x7fc01414d5d0 2024-04-25T11:42:08.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.980+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc00401a070 con 0x7fc01414d5d0 2024-04-25T11:42:08.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.980+0000 7fc00a7fc640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 0x7fbff8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:08.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.980+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7fc00408f4e0 con 0x7fc01414d5d0 2024-04-25T11:42:08.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.980+0000 7fc019efa640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 0x7fbff8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:08.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.980+0000 7fc019efa640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 0x7fbff8071910 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc0140521a0 tx=0x7fc00c009230 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:08.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:08.988+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc004021070 con 0x7fc01414d5d0 2024-04-25T11:42:09.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.475+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbff0005590 con 0x7fc01414d5d0 2024-04-25T11:42:09.461 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.475+0000 7fc00a7fc640 1 -- 172.21.15.136:0/1990741174 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6103 (secure 0 0 0) 0x7fc00405bba0 con 0x7fc01414d5d0 2024-04-25T11:42:09.461 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:09.462 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:05.650156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:09.462 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 87 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 msgr2=0x7fbff8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 0x7fbff8071910 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc0140521a0 tx=0x7fc00c009230 comp rx=0 tx=0).stop 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 msgr2=0x7fc014147370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 0x7fc014147370 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc00400f5c0 tx=0x7fc00400fa80 comp rx=0 tx=0).stop 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 shutdown_connections 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbff806f400 0x7fbff8071910 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc014156980 0x7fc0141f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc01414d5d0 0x7fc014147370 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:09.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 --2- 172.21.15.136:0/1990741174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc0000053b0 0x7fc014149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:09.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 >> 172.21.15.136:0/1990741174 conn(0x7fc01406d780 msgr2=0x7fc01406db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:09.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 shutdown_connections 2024-04-25T11:42:09.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:09.479+0000 7fbff7fff640 1 -- 172.21.15.136:0/1990741174 wait complete. 2024-04-25T11:42:09.509 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:11.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:11.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 Processor -- start 2024-04-25T11:42:11.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 -- start start 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7a4415c0d0 0x7f7a440cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a44160370 0x7f7a44166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4415c6f0 con 0x7f7a4415ff60 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7a440c6310 con 0x7f7a44160370 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a489d1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7a4415c990 con 0x7f7a4415c0d0 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a44160370 0x7f7a44166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a44160370 0x7f7a44166a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45342/0 (socket says 172.21.15.136:45342) 2024-04-25T11:42:11.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 -- 172.21.15.136:0/1228069119 learned_addr learned my addr 172.21.15.136:0/1228069119 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:11.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a4259c640 1 --2- 172.21.15.136:0/1228069119 >> v2:172.21.15.157:3300/0 conn(0x7f7a4415c0d0 0x7f7a440cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 -- 172.21.15.136:0/1228069119 >> v2:172.21.15.157:3300/0 conn(0x7f7a4415c0d0 msgr2=0x7f7a440cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:11.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 --2- 172.21.15.136:0/1228069119 >> v2:172.21.15.157:3300/0 conn(0x7f7a4415c0d0 0x7f7a440cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 -- 172.21.15.136:0/1228069119 >> v1:172.21.15.136:6789/0 conn(0x7f7a4415ff60 legacy=0x7f7a440ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:11.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.727+0000 7f7a42d9d640 1 -- 172.21.15.136:0/1228069119 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4415cc30 con 0x7f7a44160370 2024-04-25T11:42:11.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a42d9d640 1 --2- 172.21.15.136:0/1228069119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a44160370 0x7f7a44166a50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7a440734b0 tx=0x7f7a3c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1a9211226d9d7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:11.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a41d9b640 1 -- 172.21.15.136:0/1228069119 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c0b4070 con 0x7f7a44160370 2024-04-25T11:42:11.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a41d9b640 1 -- 172.21.15.136:0/1228069119 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3c0af400 con 0x7f7a44160370 2024-04-25T11:42:11.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a41d9b640 1 -- 172.21.15.136:0/1228069119 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c0af7d0 con 0x7f7a44160370 2024-04-25T11:42:11.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 -- 172.21.15.136:0/1228069119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a44160370 msgr2=0x7f7a44166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:11.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1228069119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a44160370 0x7f7a44166a50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7a440734b0 tx=0x7f7a3c01f1e0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 -- 172.21.15.136:0/1228069119 shutdown_connections 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1228069119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a44160370 0x7f7a44166a50 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1228069119 >> v2:172.21.15.157:3300/0 conn(0x7f7a4415c0d0 0x7f7a440cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 -- 172.21.15.136:0/1228069119 >> 172.21.15.136:0/1228069119 conn(0x7f7a4406f9d0 msgr2=0x7f7a4406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.731+0000 7f7a489d1640 1 -- 172.21.15.136:0/1228069119 shutdown_connections 2024-04-25T11:42:11.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 -- 172.21.15.136:0/1228069119 wait complete. 2024-04-25T11:42:11.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 Processor -- start 2024-04-25T11:42:11.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 -- start start 2024-04-25T11:42:11.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a44160370 0x7f7a441e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7a440cbc60 con 0x7f7a4415ff60 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4415fa40 con 0x7f7a4415c0d0 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a489d1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4415cf10 con 0x7f7a44160370 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45348/0 (socket says 172.21.15.136:45348) 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a42d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a44160370 0x7f7a441e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a4259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:38944/0 (socket says 172.21.15.136:38944) 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.735+0000 7f7a4259c640 1 -- 172.21.15.136:0/1206325307 learned_addr learned my addr 172.21.15.136:0/1206325307 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 -- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a44160370 msgr2=0x7f7a441e7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:11.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a44160370 0x7f7a441e7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 -- 172.21.15.136:0/1206325307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 msgr2=0x7f7a4415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 -- 172.21.15.136:0/1206325307 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a441f6a40 con 0x7f7a4415c0d0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3bfff640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a4259c640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e8d0 tx=0x7f7a3400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34017070 con 0x7f7a4415c0d0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3400c940 con 0x7f7a4415c0d0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a440cd700 con 0x7f7a4415c0d0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400ccf0 con 0x7f7a4415c0d0 2024-04-25T11:42:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a440cdca0 con 0x7f7a4415c0d0 2024-04-25T11:42:11.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a4406a350 con 0x7f7a4415c0d0 2024-04-25T11:42:11.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7a3401a070 con 0x7f7a4415c0d0 2024-04-25T11:42:11.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.739+0000 7f7a3affd640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 0x7f7a24071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:11.726 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.743+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f7a3408f7c0 con 0x7f7a4415c0d0 2024-04-25T11:42:11.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.743+0000 7f7a3bfff640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 0x7f7a24071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:11.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.743+0000 7f7a3bfff640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 0x7f7a24071910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a44146d70 tx=0x7f7a2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:11.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:11.751+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7a3405b970 con 0x7f7a4415c0d0 2024-04-25T11:42:12.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.023+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a441599f0 con 0x7f7a4415c0d0 2024-04-25T11:42:12.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a3affd640 1 -- 172.21.15.136:0/1206325307 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6103 (secure 0 0 0) 0x7f7a3405fc70 con 0x7f7a4415c0d0 2024-04-25T11:42:12.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:12.012 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:05.650156+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:12.012 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 87 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 msgr2=0x7f7a24071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 0x7f7a24071910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a44146d70 tx=0x7f7a2c009040 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 msgr2=0x7f7a441f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400e8d0 tx=0x7f7a3400ecd0 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 shutdown_connections 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7a2406f400 0x7f7a24071910 secure :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7a44146d70 tx=0x7f7a2c009040 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a44160370 0x7f7a441e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a4415ff60 0x7f7a4415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 --2- 172.21.15.136:0/1206325307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a4415c0d0 0x7f7a441f6500 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.027+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 >> 172.21.15.136:0/1206325307 conn(0x7f7a4406f9d0 msgr2=0x7f7a4406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.031+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 shutdown_connections 2024-04-25T11:42:12.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:12.031+0000 7f7a489d1640 1 -- 172.21.15.136:0/1206325307 wait complete. 2024-04-25T11:42:12.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:14.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:14.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 Processor -- start 2024-04-25T11:42:14.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 -- start start 2024-04-25T11:42:14.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe1800ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.490 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1800cbad0 con 0x7fe1800c8cd0 2024-04-25T11:42:14.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1800c7c20 con 0x7fe1800ca010 2024-04-25T11:42:14.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe18862f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe1800c8550 con 0x7fe1800ce260 2024-04-25T11:42:14.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe1863cb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe1800c8cd0 0x7fe18014f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47766/0 (socket says 172.21.15.136:47766) 2024-04-25T11:42:14.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe1863cb640 1 -- 172.21.15.136:0/2544550414 learned_addr learned my addr 172.21.15.136:0/2544550414 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:14.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.503+0000 7fe185bca640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe1800ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151731292 0 0) 0x7fe1800cbad0 con 0x7fe1800c8cd0 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe16c003530 con 0x7fe1800c8cd0 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303027695 0 0) 0x7fe1800c7c20 con 0x7fe1800ca010 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1800cbad0 con 0x7fe1800ca010 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 822102417 0 0) 0x7fe16c003530 con 0x7fe1800c8cd0 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 msgr2=0x7fe1800ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe1800ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 >> v1:172.21.15.157:6789/0 conn(0x7fe1800ca010 legacy=0x7fe1800ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1800c7f00 con 0x7fe1800c8cd0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fe174001a00 con 0x7fe1800c8cd0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 >> v1:172.21.15.136:6789/0 conn(0x7fe1800c8cd0 legacy=0x7fe18014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe16c003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe16c0040e0 0x7fe16c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe16c004d50 0x7fe16c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1800c8550 con 0x7fe16c0040e0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c00dd40 con 0x7fe16c004d50 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c00df40 con 0x7fe1800ce260 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe16c0040e0 0x7fe16c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1863cb640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe16c003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe186bcc640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe16c004d50 0x7fe16c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 -- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 msgr2=0x7fe16c003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe1800ce260 0x7fe16c003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 -- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe16c004d50 msgr2=0x7fe16c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:14.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe16c004d50 0x7fe16c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 -- 172.21.15.136:0/2544550414 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c003530 con 0x7fe16c0040e0 2024-04-25T11:42:14.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe186bcc640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe16c004d50 0x7fe16c005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe185bca640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe16c0040e0 0x7fe16c004810 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe178009ae0 tx=0x7fe17801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f146bf2ef48525 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1780b2070 con 0x7fe16c0040e0 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1780a5570 con 0x7fe16c0040e0 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.507+0000 7fe1853c9640 1 -- 172.21.15.136:0/2544550414 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1780096b0 con 0x7fe16c0040e0 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- 172.21.15.136:0/2544550414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe16c0040e0 msgr2=0x7fe16c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 --2- 172.21.15.136:0/2544550414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe16c0040e0 0x7fe16c004810 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe178009ae0 tx=0x7fe17801ca90 comp rx=0 tx=0).stop 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe186bcc640 1 -- 172.21.15.136:0/2544550414 reap_dead start 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- 172.21.15.136:0/2544550414 shutdown_connections 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- 172.21.15.136:0/2544550414 >> 172.21.15.136:0/2544550414 conn(0x7fe18006f9d0 msgr2=0x7fe18006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- 172.21.15.136:0/2544550414 shutdown_connections 2024-04-25T11:42:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- 172.21.15.136:0/2544550414 wait complete. 2024-04-25T11:42:14.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 Processor -- start 2024-04-25T11:42:14.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.511+0000 7fe18862f640 1 -- start start 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe16c0040e0 0x7fe1801f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18006cce0 0x7fe180148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1800cb720 con 0x7fe1801ee210 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe1800c91e0 con 0x7fe18006cce0 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe18006e140 con 0x7fe16c0040e0 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45374/0 (socket says 172.21.15.136:45374) 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 -- 172.21.15.136:0/2935894937 learned_addr learned my addr 172.21.15.136:0/2935894937 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe185bca640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18006cce0 0x7fe180148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe1863cb640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe16c0040e0 0x7fe1801f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 -- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe16c0040e0 msgr2=0x7fe1801f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe16c0040e0 0x7fe1801f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 -- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18006cce0 msgr2=0x7fe180148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18006cce0 0x7fe180148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 -- 172.21.15.136:0/2935894937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1801f1cf0 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe186bcc640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe18006f9a0 tx=0x7fe17c00d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c018070 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17c0067e0 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1801edbc0 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c006bd0 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1801f12b0 con 0x7fe1801ee210 2024-04-25T11:42:14.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18006a350 con 0x7fe1801ee210 2024-04-25T11:42:14.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.515+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe17c01b070 con 0x7fe1801ee210 2024-04-25T11:42:14.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.519+0000 7fe172ffd640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 0x7fe164071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:14.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.519+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7fe17c08e970 con 0x7fe1801ee210 2024-04-25T11:42:14.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.519+0000 7fe1863cb640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 0x7fe164071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:14.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.519+0000 7fe1863cb640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 0x7fe164071940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe1740012f0 tx=0x7fe174001260 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:14.507 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.523+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe17c05ab20 con 0x7fe1801ee210 2024-04-25T11:42:14.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.707+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe180149550 con 0x7fe1801ee210 2024-04-25T11:42:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe172ffd640 1 -- 172.21.15.136:0/2935894937 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+6105 (secure 0 0 0) 0x7fe17c063160 con 0x7fe1801ee210 2024-04-25T11:42:14.695 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:14.695 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:11.467255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:42:14.696 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 88 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 msgr2=0x7fe164071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 0x7fe164071940 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe1740012f0 tx=0x7fe174001260 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 msgr2=0x7fe1801ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fe18006f9a0 tx=0x7fe17c00d7d0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 shutdown_connections 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1801ee210 0x7fe1801ee620 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe18006cce0 0x7fe180148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe16406f430 0x7fe164071940 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 --2- 172.21.15.136:0/2935894937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe16c0040e0 0x7fe1801f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:14.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 >> 172.21.15.136:0/2935894937 conn(0x7fe18006f9d0 msgr2=0x7fe180071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:14.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 shutdown_connections 2024-04-25T11:42:14.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:14.711+0000 7fe18862f640 1 -- 172.21.15.136:0/2935894937 wait complete. 2024-04-25T11:42:14.851 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5449": { "addr": "172.21.15.157:6839/3020784477", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3020784477, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3020784477, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5449, "incarnation": 56, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 37 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:42:11.467255+0000", "qdb_cluster": [ 5449, 5839, 5517, 6246 ], "qdb_leader": 5449, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5449, "mds_2": 5517, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:42:16.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:17.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 Processor -- start 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 -- start start 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f910016cc40 0x7f9100165e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9100166390 0x7f9100073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f91000737a0 0x7f910006fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f91001667a0 con 0x7f9100166390 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f910016d050 con 0x7f91000737a0 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f91065be640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f910008d2b0 con 0x7f910016cc40 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f90ff7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9100166390 0x7f9100073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f90ff7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9100166390 0x7f9100073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38038/0 (socket says 172.21.15.136:38038) 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f90ff7fe640 1 -- 172.21.15.136:0/1718807417 learned_addr learned my addr 172.21.15.136:0/1718807417 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f9104b5b640 1 --2- 172.21.15.136:0/1718807417 >> v2:172.21.15.157:3300/0 conn(0x7f91000737a0 0x7f910006fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.027+0000 7f90fffff640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f910016cc40 0x7f9100165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 -- 172.21.15.136:0/1718807417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f910016cc40 msgr2=0x7f9100165e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f910016cc40 0x7f9100165e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 -- 172.21.15.136:0/1718807417 >> v2:172.21.15.157:3300/0 conn(0x7f91000737a0 msgr2=0x7f910006fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:17.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1718807417 >> v2:172.21.15.157:3300/0 conn(0x7f91000737a0 0x7f910006fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 -- 172.21.15.136:0/1718807417 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910008d560 con 0x7f9100166390 2024-04-25T11:42:17.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9100166390 0x7f9100073260 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f91000720d0 tx=0x7f90e8027a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a67e6ac8d20850 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:17.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.047+0000 7f90feffd640 1 -- 172.21.15.136:0/1718807417 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8009b30 con 0x7f9100166390 2024-04-25T11:42:17.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f90feffd640 1 -- 172.21.15.136:0/1718807417 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e8009cd0 con 0x7f9100166390 2024-04-25T11:42:17.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f90feffd640 1 -- 172.21.15.136:0/1718807417 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e800b2f0 con 0x7f9100166390 2024-04-25T11:42:17.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- 172.21.15.136:0/1718807417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9100166390 msgr2=0x7f9100073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:17.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9100166390 0x7f9100073260 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f91000720d0 tx=0x7f90e8027a00 comp rx=0 tx=0).stop 2024-04-25T11:42:17.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- 172.21.15.136:0/1718807417 shutdown_connections 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9100166390 0x7f9100073260 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 --2- 172.21.15.136:0/1718807417 >> v2:172.21.15.157:3300/0 conn(0x7f91000737a0 0x7f910006fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 --2- 172.21.15.136:0/1718807417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f910016cc40 0x7f9100165e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- 172.21.15.136:0/1718807417 >> 172.21.15.136:0/1718807417 conn(0x7f91000c9d50 msgr2=0x7f91000ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- 172.21.15.136:0/1718807417 shutdown_connections 2024-04-25T11:42:17.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- 172.21.15.136:0/1718807417 wait complete. 2024-04-25T11:42:17.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 Processor -- start 2024-04-25T11:42:17.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.051+0000 7f91065be640 1 -- start start 2024-04-25T11:42:17.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9100166390 0x7f910015ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f910016cc40 0x7f91001604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9100072a60 con 0x7f91000737a0 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f910008db20 con 0x7f910016cc40 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9100072d40 con 0x7f9100166390 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38046/0 (socket says 172.21.15.136:38046) 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 -- 172.21.15.136:0/1197103193 learned_addr learned my addr 172.21.15.136:0/1197103193 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9100166390 0x7f910015ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f9104b5b640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f910016cc40 0x7f91001604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 -- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9100166390 msgr2=0x7f910015ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9100166390 0x7f910015ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 -- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f910016cc40 msgr2=0x7f91001604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f910016cc40 0x7f91001604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 -- 172.21.15.136:0/1197103193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910016adb0 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fffff640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f90f00069e0 tx=0x7f90f0009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0018020 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f000e940 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910016be40 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f000ed10 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9100162e80 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910006a350 con 0x7f91000737a0 2024-04-25T11:42:17.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f90f001b020 con 0x7f91000737a0 2024-04-25T11:42:17.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.055+0000 7f90fcff9640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 0x7f90e0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:17.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.059+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f90f0090390 con 0x7f91000737a0 2024-04-25T11:42:17.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.059+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 0x7f90e0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:17.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.059+0000 7f90ff7fe640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 0x7f90e0071940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f90e80ba000 tx=0x7f90e80b3070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:17.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.063+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f90f0022020 con 0x7f91000737a0 2024-04-25T11:42:17.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.259+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9100075c70 con 0x7f91000737a0 2024-04-25T11:42:17.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.259+0000 7f90fcff9640 1 -- 172.21.15.136:0/1197103193 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+6105 (secure 0 0 0) 0x7f90f005cb00 con 0x7f91000737a0 2024-04-25T11:42:17.247 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:17.247 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:11.467255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449,5839,5517,6246]},"id":1}]} 2024-04-25T11:42:17.247 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 88 2024-04-25T11:42:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 msgr2=0x7f90e0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 0x7f90e0071940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f90e80ba000 tx=0x7f90e80b3070 comp rx=0 tx=0).stop 2024-04-25T11:42:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 msgr2=0x7f910016a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f90f00069e0 tx=0x7f90f0009b40 comp rx=0 tx=0).stop 2024-04-25T11:42:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 shutdown_connections 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f910016cc40 0x7f91001604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f90e006f430 0x7f90e0071940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9100166390 0x7f910015ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 --2- 172.21.15.136:0/1197103193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91000737a0 0x7f910016a870 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 >> 172.21.15.136:0/1197103193 conn(0x7f91000c9d50 msgr2=0x7f91001643f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 shutdown_connections 2024-04-25T11:42:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:17.263+0000 7f91065be640 1 -- 172.21.15.136:0/1197103193 wait complete. 2024-04-25T11:42:17.270 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:19.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:19.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 Processor -- start 2024-04-25T11:42:19.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 -- start start 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 0x7f311815cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31180c6470 con 0x7f31180c66e0 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3118156790 con 0x7f31180cdee0 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f311ddba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f31180cd390 con 0x7f3118156380 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31177fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f31180c66e0 0x7f311815f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:45128/0 (socket says 172.21.15.136:45128) 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31177fe640 1 -- 172.21.15.136:0/3425669397 learned_addr learned my addr 172.21.15.136:0/3425669397 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 0x7f311815cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373322541 0 0) 0x7f31180c6470 con 0x7f31180c66e0 2024-04-25T11:42:19.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3104003530 con 0x7f31180c66e0 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 440561314 0 0) 0x7f3104003530 con 0x7f31180c66e0 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 msgr2=0x7f311815cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 0x7f311815cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 >> v1:172.21.15.157:6789/0 conn(0x7f31180cdee0 legacy=0x7f3118166a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311815c0b0 con 0x7f31180c66e0 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f31080016a0 con 0x7f31180c66e0 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 >> v1:172.21.15.136:6789/0 conn(0x7f31180c66e0 legacy=0x7f311815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 0x7f31040039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3104003f70 0x7f3104004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104004d50 0x7f3104005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f310400bb60 con 0x7f3104003f70 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f310400bce0 con 0x7f3104004d50 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f310400bee0 con 0x7f3118156380 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3104003f70 0x7f3104004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3117fff640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104004d50 0x7f3104005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 -- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 msgr2=0x7f31040039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3118156380 0x7f31040039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 -- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104004d50 msgr2=0x7f3104005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104004d50 0x7f3104005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.447+0000 7f3116ffd640 1 -- 172.21.15.136:0/3425669397 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3104003530 con 0x7f3104003f70 2024-04-25T11:42:19.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3104003f70 0x7f3104004810 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3100009f90 tx=0x7f310001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852d9314d17255a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:19.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000b2070 con 0x7f3104003f70 2024-04-25T11:42:19.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31000a5c90 con 0x7f3104003f70 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f31167fc640 1 -- 172.21.15.136:0/3425669397 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100009c10 con 0x7f3104003f70 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 -- 172.21.15.136:0/3425669397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3104003f70 msgr2=0x7f3104004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 --2- 172.21.15.136:0/3425669397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3104003f70 0x7f3104004810 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f3100009f90 tx=0x7f310001cc80 comp rx=0 tx=0).stop 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f3117fff640 1 -- 172.21.15.136:0/3425669397 reap_dead start 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 -- 172.21.15.136:0/3425669397 shutdown_connections 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 -- 172.21.15.136:0/3425669397 >> 172.21.15.136:0/3425669397 conn(0x7f311806d6c0 msgr2=0x7f311806e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 -- 172.21.15.136:0/3425669397 shutdown_connections 2024-04-25T11:42:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.451+0000 7f311ddba640 1 -- 172.21.15.136:0/3425669397 wait complete. 2024-04-25T11:42:19.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 Processor -- start 2024-04-25T11:42:19.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- start start 2024-04-25T11:42:19.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104003f70 0x7f3118159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f311815dc30 0x7f3118148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f31181601c0 con 0x7f3118156eb0 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f311815c6e0 con 0x7f3104003f70 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f31181604a0 con 0x7f311815dc30 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38064/0 (socket says 172.21.15.136:38064) 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 -- 172.21.15.136:0/3210573343 learned_addr learned my addr 172.21.15.136:0/3210573343 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f31177fe640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104003f70 0x7f3118159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3117fff640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f311815dc30 0x7f3118148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 -- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f311815dc30 msgr2=0x7f3118148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f311815dc30 0x7f3118148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 -- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104003f70 msgr2=0x7f3118159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104003f70 0x7f3118159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 -- 172.21.15.136:0/3210573343 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31181573b0 con 0x7f3118156eb0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f3116ffd640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f310001cab0 tx=0x7f3100009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000bc070 con 0x7f3118156eb0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31000ada90 con 0x7f3118156eb0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31000b2040 con 0x7f3118156eb0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3118159770 con 0x7f3118156eb0 2024-04-25T11:42:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311815eec0 con 0x7f3118156eb0 2024-04-25T11:42:19.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.455+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311806a350 con 0x7f3118156eb0 2024-04-25T11:42:19.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.459+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f31000b63f0 con 0x7f3118156eb0 2024-04-25T11:42:19.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.459+0000 7f310ffff640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 0x7f30f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:19.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.459+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f3100129b50 con 0x7f3118156eb0 2024-04-25T11:42:19.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.459+0000 7f31177fe640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 0x7f30f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:19.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.459+0000 7f31177fe640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 0x7f30f8071910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3108004880 tx=0x7f310800c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:19.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.463+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3100016990 con 0x7f3118156eb0 2024-04-25T11:42:19.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f311815f580 con 0x7f3118156eb0 2024-04-25T11:42:19.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f310ffff640 1 -- 172.21.15.136:0/3210573343 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6102 (secure 0 0 0) 0x7f31000f5be0 con 0x7f3118156eb0 2024-04-25T11:42:19.641 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:19.641 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:17.455328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:19.641 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 89 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 msgr2=0x7f30f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 0x7f30f8071910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3108004880 tx=0x7f310800c040 comp rx=0 tx=0).stop 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 msgr2=0x7f311815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.655+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f310001cab0 tx=0x7f3100009750 comp rx=0 tx=0).stop 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 shutdown_connections 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f30f806f400 0x7f30f8071910 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f311815dc30 0x7f3118148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3118156eb0 0x7f311815d6f0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 --2- 172.21.15.136:0/3210573343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3104003f70 0x7f3118159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:19.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 >> 172.21.15.136:0/3210573343 conn(0x7f311806d6c0 msgr2=0x7f3118155110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:19.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 shutdown_connections 2024-04-25T11:42:19.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:19.659+0000 7f311ddba640 1 -- 172.21.15.136:0/3210573343 wait complete. 2024-04-25T11:42:19.661 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:21.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:21.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 Processor -- start 2024-04-25T11:42:21.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 -- start start 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b54156380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3b5415cb70 0x7f3b5415f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3b540cdee0 0x7f3b540ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3b540cd390 con 0x7f3b5415cb70 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3b5415c6e0 con 0x7f3b540cdee0 2024-04-25T11:42:21.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b5a339640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540c6470 con 0x7f3b540c66e0 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3b5415cb70 0x7f3b5415f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3b5415cb70 0x7f3b5415f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38070/0 (socket says 172.21.15.136:38070) 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 -- 172.21.15.136:0/708398857 learned_addr learned my addr 172.21.15.136:0/708398857 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b53fff640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b54156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b588d6640 1 --2- 172.21.15.136:0/708398857 >> v2:172.21.15.157:3300/0 conn(0x7f3b540cdee0 0x7f3b540ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 -- 172.21.15.136:0/708398857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 msgr2=0x7f3b54156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b54156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 -- 172.21.15.136:0/708398857 >> v2:172.21.15.157:3300/0 conn(0x7f3b540cdee0 msgr2=0x7f3b540ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 --2- 172.21.15.136:0/708398857 >> v2:172.21.15.157:3300/0 conn(0x7f3b540cdee0 0x7f3b540ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.843+0000 7f3b537fe640 1 -- 172.21.15.136:0/708398857 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5415c0b0 con 0x7f3b5415cb70 2024-04-25T11:42:21.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.847+0000 7f3b537fe640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b5415cb70 0x7f3b5415f6a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406eea0 tx=0x7f3b4801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736209c70559671f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.847+0000 7f3b52ffd640 1 -- 172.21.15.136:0/708398857 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480b5070 con 0x7f3b5415cb70 2024-04-25T11:42:21.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.847+0000 7f3b52ffd640 1 -- 172.21.15.136:0/708398857 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b480a89d0 con 0x7f3b5415cb70 2024-04-25T11:42:21.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.847+0000 7f3b52ffd640 1 -- 172.21.15.136:0/708398857 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480a8da0 con 0x7f3b5415cb70 2024-04-25T11:42:21.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- 172.21.15.136:0/708398857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b5415cb70 msgr2=0x7f3b5415f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b5415cb70 0x7f3b5415f6a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f3b5406eea0 tx=0x7f3b4801ed10 comp rx=0 tx=0).stop 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- 172.21.15.136:0/708398857 shutdown_connections 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b5415cb70 0x7f3b5415f6a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- 172.21.15.136:0/708398857 >> v2:172.21.15.157:3300/0 conn(0x7f3b540cdee0 0x7f3b540ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- 172.21.15.136:0/708398857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b54156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- 172.21.15.136:0/708398857 >> 172.21.15.136:0/708398857 conn(0x7f3b5406d6c0 msgr2=0x7f3b5406e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- 172.21.15.136:0/708398857 shutdown_connections 2024-04-25T11:42:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- 172.21.15.136:0/708398857 wait complete. 2024-04-25T11:42:21.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 Processor -- start 2024-04-25T11:42:21.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- start start 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b541f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b5415cb70 0x7f3b5414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3b541604a0 con 0x7f3b540cdee0 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3b541601c0 con 0x7f3b5415cb70 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.851+0000 7f3b5a339640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3b5415c370 con 0x7f3b540c66e0 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38074/0 (socket says 172.21.15.136:38074) 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 -- 172.21.15.136:0/160777014 learned_addr learned my addr 172.21.15.136:0/160777014 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b53fff640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b541f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b588d6640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b5415cb70 0x7f3b5414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 -- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 msgr2=0x7f3b541f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b541f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 -- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b5415cb70 msgr2=0x7f3b5414f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b5415cb70 0x7f3b5414f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 -- 172.21.15.136:0/160777014 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b541f3780 con 0x7f3b540cdee0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b537fe640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3b4801d3c0 tx=0x7f3b4801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480b5070 con 0x7f3b540cdee0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b480b0a50 con 0x7f3b540cdee0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5414e660 con 0x7f3b540cdee0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480b0e20 con 0x7f3b540cdee0 2024-04-25T11:42:21.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541e6ec0 con 0x7f3b540cdee0 2024-04-25T11:42:21.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5406a350 con 0x7f3b540cdee0 2024-04-25T11:42:21.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3b480c2060 con 0x7f3b540cdee0 2024-04-25T11:42:21.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 0x7f3b40071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:21.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f3b4812cb10 con 0x7f3b540cdee0 2024-04-25T11:42:21.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.855+0000 7f3b53fff640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 0x7f3b40071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:21.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.859+0000 7f3b53fff640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 0x7f3b40071910 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b440045f0 tx=0x7f3b44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:21.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:21.863+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3b48018870 con 0x7f3b540cdee0 2024-04-25T11:42:22.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.063+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b540b4990 con 0x7f3b540cdee0 2024-04-25T11:42:22.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.063+0000 7f3b50ff9640 1 -- 172.21.15.136:0/160777014 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+6102 (secure 0 0 0) 0x7f3b480f9280 con 0x7f3b540cdee0 2024-04-25T11:42:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:22.049 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:17.455328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[6246,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:22.050 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 89 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 msgr2=0x7f3b40071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 0x7f3b40071910 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b440045f0 tx=0x7f3b44008040 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 msgr2=0x7f3b5414e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f3b4801d3c0 tx=0x7f3b4801bab0 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 shutdown_connections 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b5415cb70 0x7f3b5414f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b540cdee0 0x7f3b5414e120 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3b4006f400 0x7f3b40071910 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 --2- 172.21.15.136:0/160777014 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b540c66e0 0x7f3b541f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 >> 172.21.15.136:0/160777014 conn(0x7f3b5406d6c0 msgr2=0x7f3b540ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 shutdown_connections 2024-04-25T11:42:22.052 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:22.067+0000 7f3b5a339640 1 -- 172.21.15.136:0/160777014 wait complete. 2024-04-25T11:42:22.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:42:24.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:24.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 Processor -- start 2024-04-25T11:42:24.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 -- start start 2024-04-25T11:42:24.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7c640729c0 0x7f7c6415ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 0x7f7c640c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c641605b0 con 0x7f7c640c8ef0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f7c6415c820 con 0x7f7c640729c0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c6a162640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7c64072dd0 con 0x7f7c64156210 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c637fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7c640729c0 0x7f7c6415ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c63fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7c640c8ef0 0x7f7c6406fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:45138/0 (socket says 172.21.15.136:45138) 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c63fff640 1 -- 172.21.15.136:0/2393583814 learned_addr learned my addr 172.21.15.136:0/2393583814 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c62ffd640 1 --2- 172.21.15.136:0/2393583814 >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 0x7f7c640c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c637fe640 1 -- 172.21.15.136:0/2393583814 >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 msgr2=0x7f7c640c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c637fe640 1 --2- 172.21.15.136:0/2393583814 >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 0x7f7c640c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c637fe640 1 -- 172.21.15.136:0/2393583814 >> v1:172.21.15.136:6789/0 conn(0x7f7c640c8ef0 legacy=0x7f7c6406fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c637fe640 1 -- 172.21.15.136:0/2393583814 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c641602d0 con 0x7f7c640729c0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c627fc640 1 -- 172.21.15.136:0/2393583814 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946699473 0 0) 0x7f7c641605b0 con 0x7f7c640c8ef0 2024-04-25T11:42:24.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.247+0000 7f7c62ffd640 1 --2- 172.21.15.136:0/2393583814 >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 0x7f7c640c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:24.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.251+0000 7f7c637fe640 1 --2- 172.21.15.136:0/2393583814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c6415ccb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406dc30 tx=0x7f7c5c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b862226383f9a595 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:24.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.251+0000 7f7c627fc640 1 -- 172.21.15.136:0/2393583814 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0b2070 con 0x7f7c640729c0 2024-04-25T11:42:24.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.251+0000 7f7c627fc640 1 -- 172.21.15.136:0/2393583814 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c5c0a58e0 con 0x7f7c640729c0 2024-04-25T11:42:24.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.251+0000 7f7c627fc640 1 -- 172.21.15.136:0/2393583814 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a5cb0 con 0x7f7c640729c0 2024-04-25T11:42:24.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- 172.21.15.136:0/2393583814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 msgr2=0x7f7c6415ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- 172.21.15.136:0/2393583814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c6415ccb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7c6406dc30 tx=0x7f7c5c01c930 comp rx=0 tx=0).stop 2024-04-25T11:42:24.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- 172.21.15.136:0/2393583814 shutdown_connections 2024-04-25T11:42:24.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- 172.21.15.136:0/2393583814 >> v2:172.21.15.157:3300/0 conn(0x7f7c64156210 0x7f7c640c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- 172.21.15.136:0/2393583814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c6415ccb0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- 172.21.15.136:0/2393583814 >> 172.21.15.136:0/2393583814 conn(0x7f7c640c9d50 msgr2=0x7f7c640ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:24.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- 172.21.15.136:0/2393583814 shutdown_connections 2024-04-25T11:42:24.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- 172.21.15.136:0/2393583814 wait complete. 2024-04-25T11:42:24.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 Processor -- start 2024-04-25T11:42:24.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- start start 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c640c8ef0 0x7f7c640c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c64156210 0x7f7c640c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7c64156620 con 0x7f7c640729c0 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7c640c9710 con 0x7f7c64156210 2024-04-25T11:42:24.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c6a162640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7c641567a0 con 0x7f7c640c8ef0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c637fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c637fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38096/0 (socket says 172.21.15.136:38096) 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.255+0000 7f7c637fe640 1 -- 172.21.15.136:0/4085931386 learned_addr learned my addr 172.21.15.136:0/4085931386 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 -- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c640c8ef0 msgr2=0x7f7c640c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c640c8ef0 0x7f7c640c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 -- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c64156210 msgr2=0x7f7c640c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c64156210 0x7f7c640c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 -- 172.21.15.136:0/4085931386 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c640cb050 con 0x7f7c640729c0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c637fe640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c019540 tx=0x7f7c5c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0b2050 con 0x7f7c640729c0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c5c0be070 con 0x7f7c640729c0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0ada50 con 0x7f7c640729c0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6415ec90 con 0x7f7c640729c0 2024-04-25T11:42:24.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6415e9b0 con 0x7f7c640729c0 2024-04-25T11:42:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6406a350 con 0x7f7c640729c0 2024-04-25T11:42:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7c5c0adbf0 con 0x7f7c640729c0 2024-04-25T11:42:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 0x7f7c48071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:24.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f7c5c12ac70 con 0x7f7c640729c0 2024-04-25T11:42:24.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.259+0000 7f7c62ffd640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 0x7f7c48071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.263+0000 7f7c62ffd640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 0x7f7c48071910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7c6415a430 tx=0x7f7c4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:24.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.263+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7c5c016520 con 0x7f7c640729c0 2024-04-25T11:42:24.451 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.467+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c64159ae0 con 0x7f7c640729c0 2024-04-25T11:42:24.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.467+0000 7f7c5bfff640 1 -- 172.21.15.136:0/4085931386 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+6107 (secure 0 0 0) 0x7f7c5c0f7470 con 0x7f7c640729c0 2024-04-25T11:42:24.453 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:24.453 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5449,5839,5517,5911,6246]},"id":1}]} 2024-04-25T11:42:24.453 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 90 2024-04-25T11:42:24.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 msgr2=0x7f7c48071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 0x7f7c48071910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7c6415a430 tx=0x7f7c4c008040 comp rx=0 tx=0).stop 2024-04-25T11:42:24.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 msgr2=0x7f7c64140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:24.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c019540 tx=0x7f7c5c0196d0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 shutdown_connections 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c64156210 0x7f7c640c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7c4806f400 0x7f7c48071910 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c640c8ef0 0x7f7c640c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 --2- 172.21.15.136:0/4085931386 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c640729c0 0x7f7c64140da0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 >> 172.21.15.136:0/4085931386 conn(0x7f7c640c9d50 msgr2=0x7f7c64155200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 shutdown_connections 2024-04-25T11:42:24.456 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:24.471+0000 7f7c6a162640 1 -- 172.21.15.136:0/4085931386 wait complete. 2024-04-25T11:42:24.474 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5408 2024-04-25T11:42:24.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2024-04-25T11:42:25.476 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-25T11:42:25.476 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T11:42:25.476 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T11:42:25.479 INFO:tasks.ceph.mds.b:Started 2024-04-25T11:42:25.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:25.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 Processor -- start 2024-04-25T11:42:25.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 -- start start 2024-04-25T11:42:25.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fac50153190 0x7fac500c9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac50163660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac500c8090 con 0x7fac500c9570 2024-04-25T11:42:25.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fac500c8370 con 0x7fac50153190 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac5481d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac500c6aa0 con 0x7fac501594b0 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac4dd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fac500c9570 0x7fac50158da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:47308/0 (socket says 172.21.15.136:47308) 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac4dd9b640 1 -- 172.21.15.136:0/3362206986 learned_addr learned my addr 172.21.15.136:0/3362206986 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.655+0000 7fac4ed9d640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac50163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4e59c640 1 --2- 172.21.15.136:0/3362206986 >> v2:172.21.15.157:3300/0 conn(0x7fac50153190 0x7fac500c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 129595446 0 0) 0x7fac500c8090 con 0x7fac500c9570 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac38003530 con 0x7fac500c9570 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 924626025 0 0) 0x7fac38003530 con 0x7fac500c9570 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 msgr2=0x7fac50163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac50163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 >> v2:172.21.15.157:3300/0 conn(0x7fac50153190 msgr2=0x7fac500c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 --2- 172.21.15.136:0/3362206986 >> v2:172.21.15.157:3300/0 conn(0x7fac50153190 0x7fac500c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac500c6d80 con 0x7fac500c9570 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fac44001680 con 0x7fac500c9570 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 >> v1:172.21.15.136:6789/0 conn(0x7fac500c9570 legacy=0x7fac50158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac380039c0 0x7fac38003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac38004810 0x7fac38004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac38005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fac500c8370 con 0x7fac38004810 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac500c6aa0 con 0x7fac380039c0 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac3800ba50 con 0x7fac501594b0 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac38004810 0x7fac38004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4ed9d640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac38005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4e59c640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac380039c0 0x7fac38003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 -- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 msgr2=0x7fac38005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:25.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac501594b0 0x7fac38005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 -- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac380039c0 msgr2=0x7fac38003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:25.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac380039c0 0x7fac38003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 -- 172.21.15.136:0/3362206986 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac38003530 con 0x7fac38004810 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac38004810 0x7fac38004c20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac4400c240 tx=0x7fac4401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3847515ec341df7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44007d60 con 0x7fac38004810 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440a9020 con 0x7fac38004810 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.659+0000 7fac4d59a640 1 -- 172.21.15.136:0/3362206986 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5be0 con 0x7fac38004810 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- 172.21.15.136:0/3362206986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac38004810 msgr2=0x7fac38004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 --2- 172.21.15.136:0/3362206986 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac38004810 0x7fac38004c20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac4400c240 tx=0x7fac4401ccc0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4ed9d640 1 -- 172.21.15.136:0/3362206986 reap_dead start 2024-04-25T11:42:25.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- 172.21.15.136:0/3362206986 shutdown_connections 2024-04-25T11:42:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- 172.21.15.136:0/3362206986 >> 172.21.15.136:0/3362206986 conn(0x7fac500c9d50 msgr2=0x7fac500ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- 172.21.15.136:0/3362206986 shutdown_connections 2024-04-25T11:42:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- 172.21.15.136:0/3362206986 wait complete. 2024-04-25T11:42:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 Processor -- start 2024-04-25T11:42:25.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- start start 2024-04-25T11:42:25.650 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 0x7fac50149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac50146ce0 0x7fac501470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fac500ca870 con 0x7fac50156220 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac500c7340 con 0x7fac38004810 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac5481d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac500ca9f0 con 0x7fac50146ce0 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 0x7fac50149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52898/0 (socket says 172.21.15.136:52898) 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4dd9b640 1 -- 172.21.15.136:0/313337342 learned_addr learned my addr 172.21.15.136:0/313337342 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.663+0000 7fac4ed9d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac50146ce0 0x7fac501470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 -- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac50146ce0 msgr2=0x7fac501470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac50146ce0 0x7fac501470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 -- 172.21.15.136:0/313337342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 msgr2=0x7fac500cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 -- 172.21.15.136:0/313337342 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5014a1b0 con 0x7fac38004810 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4e59c640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 0x7fac50149c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fac40011590 tx=0x7fac40011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40019070 con 0x7fac38004810 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac40003620 con 0x7fac38004810 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac400039f0 con 0x7fac38004810 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac50153790 con 0x7fac38004810 2024-04-25T11:42:25.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac50149750 con 0x7fac38004810 2024-04-25T11:42:25.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5006a350 con 0x7fac38004810 2024-04-25T11:42:25.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fac4001c070 con 0x7fac38004810 2024-04-25T11:42:25.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 0x7fac30071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:25.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7fac4008f9b0 con 0x7fac38004810 2024-04-25T11:42:25.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.667+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 0x7fac30071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:25.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.671+0000 7fac4dd9b640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 0x7fac30071940 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fac4401cd00 tx=0x7fac44001ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:25.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.671+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fac4005bb40 con 0x7fac38004810 2024-04-25T11:42:25.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.863+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac50151120 con 0x7fac38004810 2024-04-25T11:42:25.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.863+0000 7fac3effd640 1 -- 172.21.15.136:0/313337342 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+6107 (secure 0 0 0) 0x7fac4005fe40 con 0x7fac38004810 2024-04-25T11:42:25.850 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:25.850 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5449,5839,5517,5911,6246]},"id":1}]} 2024-04-25T11:42:25.850 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 90 2024-04-25T11:42:25.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 msgr2=0x7fac30071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 0x7fac30071940 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fac4401cd00 tx=0x7fac44001ac0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 msgr2=0x7fac50149c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 0x7fac50149c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fac40011590 tx=0x7fac40011a50 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 shutdown_connections 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fac3006f430 0x7fac30071940 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac50146ce0 0x7fac501470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac50156220 0x7fac500cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 --2- 172.21.15.136:0/313337342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac38004810 0x7fac50149c70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 >> 172.21.15.136:0/313337342 conn(0x7fac500c9d50 msgr2=0x7fac5014fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 shutdown_connections 2024-04-25T11:42:25.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:25.867+0000 7fac5481d640 1 -- 172.21.15.136:0/313337342 wait complete. 2024-04-25T11:42:25.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T11:42:27.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:42:27.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:42:27.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:28.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.043+0000 7f05ab576640 1 Processor -- start 2024-04-25T11:42:28.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.043+0000 7f05ab576640 1 -- start start 2024-04-25T11:42:28.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05ab576640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4143170 0x7f05a415d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05ab576640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f05a4070c80 0x7f05a4164790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05ab576640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a4143580 con 0x7f05a4070870 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05ab576640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f05a415dd20 con 0x7f05a4070c80 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05ab576640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f05a415e000 con 0x7f05a4143170 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a8b11640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f05a4070870 0x7f05a40c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:47930/0 (socket says 172.21.15.136:47930) 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a8b11640 1 -- 172.21.15.136:0/2534318745 learned_addr learned my addr 172.21.15.136:0/2534318745 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9b13640 1 --2- 172.21.15.136:0/2534318745 >> v2:172.21.15.157:3300/0 conn(0x7f05a4070c80 0x7f05a4164790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9312640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4143170 0x7f05a415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f059bfff640 1 -- 172.21.15.136:0/2534318745 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411525237 0 0) 0x7f05a0001120 con 0x7f05a4070870 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f059bfff640 1 -- 172.21.15.136:0/2534318745 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0590003530 con 0x7f05a4070870 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9b13640 1 -- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4143170 msgr2=0x7f05a415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9b13640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4143170 0x7f05a415d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9b13640 1 -- 172.21.15.136:0/2534318745 >> v1:172.21.15.157:6789/0 conn(0x7f05a4070870 legacy=0x7f05a40c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.047+0000 7f05a9b13640 1 -- 172.21.15.136:0/2534318745 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a406fd90 con 0x7f05a4070c80 2024-04-25T11:42:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05a9b13640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4070c80 0x7f05a4164790 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f059c009d90 tx=0x7f059c01f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a01ed5919035e1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f059bfff640 1 -- 172.21.15.136:0/2534318745 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0b4070 con 0x7f05a4070c80 2024-04-25T11:42:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f059bfff640 1 -- 172.21.15.136:0/2534318745 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059c0af600 con 0x7f05a4070c80 2024-04-25T11:42:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f059bfff640 1 -- 172.21.15.136:0/2534318745 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0af9b0 con 0x7f05a4070c80 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 -- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4070c80 msgr2=0x7f05a4164790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4070c80 0x7f05a4164790 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f059c009d90 tx=0x7f059c01f150 comp rx=0 tx=0).stop 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 -- 172.21.15.136:0/2534318745 shutdown_connections 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4070c80 0x7f05a4164790 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 --2- 172.21.15.136:0/2534318745 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4143170 0x7f05a415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 -- 172.21.15.136:0/2534318745 >> 172.21.15.136:0/2534318745 conn(0x7f05a40c9d50 msgr2=0x7f05a40ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 -- 172.21.15.136:0/2534318745 shutdown_connections 2024-04-25T11:42:28.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.051+0000 7f05ab576640 1 -- 172.21.15.136:0/2534318745 wait complete. 2024-04-25T11:42:28.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 Processor -- start 2024-04-25T11:42:28.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- start start 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4070c80 0x7f05a41f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4143170 0x7f05a4157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f05a415e2e0 con 0x7f05a4070870 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f05a4070350 con 0x7f05a4143170 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f05a40712a0 con 0x7f05a4070c80 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52908/0 (socket says 172.21.15.136:52908) 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 -- 172.21.15.136:0/1751905741 learned_addr learned my addr 172.21.15.136:0/1751905741 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a8b11640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4070c80 0x7f05a41f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9b13640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4143170 0x7f05a4157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 -- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4070c80 msgr2=0x7f05a41f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4070c80 0x7f05a41f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 -- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4143170 msgr2=0x7f05a4157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4143170 0x7f05a4157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 -- 172.21.15.136:0/1751905741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a41f4160 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05a9312640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f059400e8a0 tx=0x7f059400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594017070 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059400c8e0 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a41f06a0 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059400cc90 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a41f0c60 con 0x7f05a4070870 2024-04-25T11:42:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.055+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a406a350 con 0x7f05a4070870 2024-04-25T11:42:28.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.059+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f059401a070 con 0x7f05a4070870 2024-04-25T11:42:28.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.059+0000 7f0599ffb640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 0x7f0588071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:28.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.059+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f059408fd00 con 0x7f05a4070870 2024-04-25T11:42:28.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.059+0000 7f05a8b11640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 0x7f0588071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:28.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.059+0000 7f05a8b11640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 0x7f0588071910 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f05a413e680 tx=0x7f05a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:28.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.063+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f059405c3e0 con 0x7f05a4070870 2024-04-25T11:42:28.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.259+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a41f3880 con 0x7f05a4070870 2024-04-25T11:42:28.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f0599ffb640 1 -- 172.21.15.136:0/1751905741 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7f059405bce0 con 0x7f05a4070870 2024-04-25T11:42:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:28.248 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:28.248 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 msgr2=0x7f0588071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 0x7f0588071910 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f05a413e680 tx=0x7f05a0009040 comp rx=0 tx=0).stop 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 msgr2=0x7f05a4158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f059400e8a0 tx=0x7f059400eca0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 shutdown_connections 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05a4143170 0x7f05a4157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f058806f400 0x7f0588071910 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05a4070c80 0x7f05a41f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 --2- 172.21.15.136:0/1751905741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05a4070870 0x7f05a4158760 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:28.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 >> 172.21.15.136:0/1751905741 conn(0x7f05a40c9d50 msgr2=0x7f05a413e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:28.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 shutdown_connections 2024-04-25T11:42:28.251 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:28.263+0000 7f05ab576640 1 -- 172.21.15.136:0/1751905741 wait complete. 2024-04-25T11:42:28.270 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-25T11:42:28.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before thrashing 2024-04-25T11:42:31.407 INFO:tasks.ceph.mds.a.smithi136.stderr:2024-04-25T11:42:31.419+0000 7f0be9165640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T11:42:31.407 INFO:tasks.ceph.mds.a.smithi136.stderr:2024-04-25T11:42:31.419+0000 7f0be9165640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T11:42:31.407 INFO:tasks.ceph.mds.a.smithi136.stderr:2024-04-25T11:42:31.419+0000 7f0be9165640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T11:42:31.407 INFO:tasks.ceph.mds.a.smithi136.stderr:2024-04-25T11:42:31.419+0000 7f0be9165640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T11:42:31.407 INFO:tasks.ceph.mds.a.smithi136.stderr:2024-04-25T11:42:31.419+0000 7f0be9165640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T11:42:37.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:37.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 Processor -- start 2024-04-25T11:42:37.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 -- start start 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d400334f0 0x7f2d40006400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d40006940 0x7f2d400e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d400df820 con 0x7f2d4004ba50 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4004c510 con 0x7f2d40006940 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.715+0000 7f2d4ea6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d400de930 con 0x7f2d400334f0 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d400334f0 0x7f2d40006400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d400334f0 0x7f2d40006400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56050/0 (socket says 172.21.15.136:56050) 2024-04-25T11:42:37.703 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 -- 172.21.15.136:0/1344523276 learned_addr learned my addr 172.21.15.136:0/1344523276 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2821673490 0 0) 0x7f2d400df820 con 0x7f2d4004ba50 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d34003530 con 0x7f2d4004ba50 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4275035057 0 0) 0x7f2d34003530 con 0x7f2d4004ba50 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d400334f0 msgr2=0x7f2d40006400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d400334f0 0x7f2d40006400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 >> v2:172.21.15.157:3300/0 conn(0x7f2d40006940 msgr2=0x7f2d400e5cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:37.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 --2- 172.21.15.136:0/1344523276 >> v2:172.21.15.157:3300/0 conn(0x7f2d40006940 0x7f2d400e5cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d400351a0 con 0x7f2d4004ba50 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f2d440016a0 con 0x7f2d4004ba50 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 >> v1:172.21.15.136:6789/0 conn(0x7f2d4004ba50 legacy=0x7f2d400df110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d340039c0 0x7f2d34003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d34004810 0x7f2d34004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34005160 0x7f2d340145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4004c510 con 0x7f2d34004810 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d400de930 con 0x7f2d340039c0 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d34009bc0 con 0x7f2d34005160 2024-04-25T11:42:37.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d34004810 0x7f2d34004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4da6a640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d340039c0 0x7f2d34003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 -- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34005160 msgr2=0x7f2d340145d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:37.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34005160 0x7f2d340145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 -- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d340039c0 msgr2=0x7f2d34003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:37.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d340039c0 0x7f2d34003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.719+0000 7f2d4d269640 1 -- 172.21.15.136:0/1344523276 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d400351a0 con 0x7f2d34004810 2024-04-25T11:42:37.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4d269640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d34004810 0x7f2d34004c20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000b70 tx=0x7f2d48099b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdc2ea243846acb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48128070 con 0x7f2d34004810 2024-04-25T11:42:37.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4ca68640 1 -- 172.21.15.136:0/1344523276 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d481229f0 con 0x7f2d34004810 2024-04-25T11:42:37.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/1344523276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d34004810 msgr2=0x7f2d34004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:37.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4ea6c640 1 --2- 172.21.15.136:0/1344523276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d34004810 0x7f2d34004c20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000b70 tx=0x7f2d48099b00 comp rx=0 tx=0).stop 2024-04-25T11:42:37.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.727+0000 7f2d4e26b640 1 -- 172.21.15.136:0/1344523276 reap_dead start 2024-04-25T11:42:37.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.731+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/1344523276 shutdown_connections 2024-04-25T11:42:37.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.731+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/1344523276 >> 172.21.15.136:0/1344523276 conn(0x7f2d400503c0 msgr2=0x7f2d400507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:37.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.731+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/1344523276 shutdown_connections 2024-04-25T11:42:37.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.731+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/1344523276 wait complete. 2024-04-25T11:42:37.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 Processor -- start 2024-04-25T11:42:37.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 -- start start 2024-04-25T11:42:37.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34004810 0x7f2d400d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.720 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 0x7f2d40049910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2d400debf0 con 0x7f2d400d24f0 2024-04-25T11:42:37.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d40035760 con 0x7f2d400cf250 2024-04-25T11:42:37.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.735+0000 7f2d4ea6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d40034ea0 con 0x7f2d34004810 2024-04-25T11:42:37.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:47964/0 (socket says 172.21.15.136:47964) 2024-04-25T11:42:37.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 -- 172.21.15.136:0/3849531538 learned_addr learned my addr 172.21.15.136:0/3849531538 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:37.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 -- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34004810 msgr2=0x7f2d400d81f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:37.723 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34004810 0x7f2d400d81f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 -- 172.21.15.136:0/3849531538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 msgr2=0x7f2d40049910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4e26b640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 0x7f2d40049910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 0x7f2d40049910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 -- 172.21.15.136:0/3849531538 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d40186cd0 con 0x7f2d400cf250 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4e26b640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 0x7f2d40049910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:37.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.739+0000 7f2d4d269640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d480515a0 tx=0x7f2d4806c290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.763+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48122e60 con 0x7f2d400cf250 2024-04-25T11:42:37.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.763+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/3849531538 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d40182bb0 con 0x7f2d400cf250 2024-04-25T11:42:37.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.763+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/3849531538 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d40183170 con 0x7f2d400cf250 2024-04-25T11:42:37.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.763+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4813b070 con 0x7f2d400cf250 2024-04-25T11:42:37.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48128040 con 0x7f2d400cf250 2024-04-25T11:42:37.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2d48133070 con 0x7f2d400cf250 2024-04-25T11:42:37.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d3e7fc640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 0x7f2d2c071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:37.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f2d481a6e40 con 0x7f2d400cf250 2024-04-25T11:42:37.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d4da6a640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 0x7f2d2c071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:37.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.767+0000 7f2d4da6a640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 0x7f2d2c071a40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2d44001810 tx=0x7f2d44001580 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.771+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/3849531538 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d300052a0 con 0x7f2d400cf250 2024-04-25T11:42:37.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:37.783+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2d480939d0 con 0x7f2d400cf250 2024-04-25T11:42:38.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.487+0000 7f2d4ea6c640 1 -- 172.21.15.136:0/3849531538 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d30005590 con 0x7f2d400cf250 2024-04-25T11:42:38.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.495+0000 7f2d3e7fc640 1 -- 172.21.15.136:0/3849531538 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7f2d48172f70 con 0x7f2d400cf250 2024-04-25T11:42:38.490 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:38.490 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:38.490 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 msgr2=0x7f2d2c071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 0x7f2d2c071a40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2d44001810 tx=0x7f2d44001580 comp rx=0 tx=0).stop 2024-04-25T11:42:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 msgr2=0x7f2d40186790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:38.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d480515a0 tx=0x7f2d4806c290 comp rx=0 tx=0).stop 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 shutdown_connections 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d400d24f0 0x7f2d40049910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d400cf250 0x7f2d40186790 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d2c06f530 0x7f2d2c071a40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 --2- 172.21.15.136:0/3849531538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d34004810 0x7f2d400d81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:38.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 >> 172.21.15.136:0/3849531538 conn(0x7f2d400503c0 msgr2=0x7f2d40005790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:38.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 shutdown_connections 2024-04-25T11:42:38.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:38.511+0000 7f2d2bfff640 1 -- 172.21.15.136:0/3849531538 wait complete. 2024-04-25T11:42:38.590 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2024-04-25T11:42:38.590 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-25T11:42:38.590 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:42:38.619 INFO:tasks.ceph.mds.f:Stopped 2024-04-25T11:42:38.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:39.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 Processor -- start 2024-04-25T11:42:39.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 -- start start 2024-04-25T11:42:39.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f81ec033520 0x7f81ec0d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 0x7f81ec0e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0dd5a0 con 0x7f81ec0062d0 2024-04-25T11:42:39.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f81ec0de0b0 con 0x7f81ec033520 2024-04-25T11:42:39.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.155+0000 7f81fa0d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec0dd810 con 0x7f81ec0066e0 2024-04-25T11:42:39.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.159+0000 7f81f37fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f81ec033520 0x7f81ec0d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.159+0000 7f81f3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 0x7f81ec0e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.159+0000 7f81f3fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 0x7f81ec0e7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56058/0 (socket says 172.21.15.136:56058) 2024-04-25T11:42:39.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.159+0000 7f81f3fff640 1 -- 172.21.15.136:0/48622628 learned_addr learned my addr 172.21.15.136:0/48622628 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:39.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.159+0000 7f81f2ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f81ec0062d0 0x7f81ec04b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50918/0 (socket says 172.21.15.136:50918) 2024-04-25T11:42:39.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.167+0000 7f81f37fe640 1 -- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 msgr2=0x7f81ec0e7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:39.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.167+0000 7f81f37fe640 1 --2- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 0x7f81ec0e7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.167+0000 7f81f37fe640 1 -- 172.21.15.136:0/48622628 >> v1:172.21.15.136:6789/0 conn(0x7f81ec0062d0 legacy=0x7f81ec04b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.167+0000 7f81f37fe640 1 -- 172.21.15.136:0/48622628 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec0ddaf0 con 0x7f81ec033520 2024-04-25T11:42:39.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.171+0000 7f81f37fe640 1 --2- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec033520 0x7f81ec0d7280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097c0 tx=0x7f81e801c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803bf687c07ee8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.183+0000 7f81f27fc640 1 -- 172.21.15.136:0/48622628 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80b0070 con 0x7f81ec033520 2024-04-25T11:42:39.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.183+0000 7f81f27fc640 1 -- 172.21.15.136:0/48622628 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e80ab600 con 0x7f81ec033520 2024-04-25T11:42:39.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.183+0000 7f81f27fc640 1 -- 172.21.15.136:0/48622628 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80ab9d0 con 0x7f81ec033520 2024-04-25T11:42:39.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.183+0000 7f81fa0d4640 1 -- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec033520 msgr2=0x7f81ec0d7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.183+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec033520 0x7f81ec0d7280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097c0 tx=0x7f81e801c330 comp rx=0 tx=0).stop 2024-04-25T11:42:39.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.187+0000 7f81fa0d4640 1 -- 172.21.15.136:0/48622628 shutdown_connections 2024-04-25T11:42:39.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.187+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec0066e0 0x7f81ec0e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.187+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/48622628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec033520 0x7f81ec0d7280 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.187+0000 7f81fa0d4640 1 -- 172.21.15.136:0/48622628 >> 172.21.15.136:0/48622628 conn(0x7f81ec0503f0 msgr2=0x7f81ec050800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.207+0000 7f81fa0d4640 1 -- 172.21.15.136:0/48622628 shutdown_connections 2024-04-25T11:42:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.207+0000 7f81fa0d4640 1 -- 172.21.15.136:0/48622628 wait complete. 2024-04-25T11:42:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.207+0000 7f81fa0d4640 1 Processor -- start 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.207+0000 7f81fa0d4640 1 -- start start 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec0066e0 0x7f81ec0cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec033520 0x7f81ec0d3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec04b180 con 0x7f81ec0062d0 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec04aea0 con 0x7f81ec0066e0 2024-04-25T11:42:39.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81fa0d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec006af0 con 0x7f81ec033520 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:35132/0 (socket says 172.21.15.136:35132) 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 -- 172.21.15.136:0/3737274228 learned_addr learned my addr 172.21.15.136:0/3737274228 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 -- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec033520 msgr2=0x7f81ec0d3270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec033520 0x7f81ec0d3270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 -- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec0066e0 msgr2=0x7f81ec0cb140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec0066e0 0x7f81ec0cb140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 -- 172.21.15.136:0/3737274228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec04ee80 con 0x7f81ec0062d0 2024-04-25T11:42:39.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.211+0000 7f81f37fe640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f81e801cbd0 tx=0x7f81e80190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.215+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80b0040 con 0x7f81ec0062d0 2024-04-25T11:42:39.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.219+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec0470a0 con 0x7f81ec0062d0 2024-04-25T11:42:39.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.219+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ec0d2c50 con 0x7f81ec0062d0 2024-04-25T11:42:39.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e80ba070 con 0x7f81ec0062d0 2024-04-25T11:42:39.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80aab50 con 0x7f81ec0062d0 2024-04-25T11:42:39.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f81e80b3070 con 0x7f81ec0062d0 2024-04-25T11:42:39.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81dffff640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 0x7f81e4071b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:39.208 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f81e8129250 con 0x7f81ec0062d0 2024-04-25T11:42:39.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81f2ffd640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 0x7f81e4071b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:39.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.223+0000 7f81f2ffd640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 0x7f81e4071b80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f81ec050be0 tx=0x7f81e0001c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:39.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.227+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ec0055b0 con 0x7f81ec0062d0 2024-04-25T11:42:39.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.239+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f81e80c3020 con 0x7f81ec0062d0 2024-04-25T11:42:39.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.895+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81ec047910 con 0x7f81ec0062d0 2024-04-25T11:42:39.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.895+0000 7f81dffff640 1 -- 172.21.15.136:0/3737274228 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7f81e8015fb0 con 0x7f81ec0062d0 2024-04-25T11:42:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:39.882 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:39.882 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:39.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 msgr2=0x7f81e4071b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.885 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 0x7f81e4071b80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f81ec050be0 tx=0x7f81e0001c00 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 msgr2=0x7f81ec04fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f81e801cbd0 tx=0x7f81e80190d0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 shutdown_connections 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f81e406f670 0x7f81e4071b80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81ec033520 0x7f81ec0d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81ec0066e0 0x7f81ec0cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 --2- 172.21.15.136:0/3737274228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81ec0062d0 0x7f81ec04fbe0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.899+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 >> 172.21.15.136:0/3737274228 conn(0x7f81ec0503f0 msgr2=0x7f81ec0dbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.903+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 shutdown_connections 2024-04-25T11:42:39.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:39.903+0000 7f81fa0d4640 1 -- 172.21.15.136:0/3737274228 wait complete. 2024-04-25T11:42:39.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:42:41.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:42.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 Processor -- start 2024-04-25T11:42:42.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 -- start start 2024-04-25T11:42:42.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5e3c0468d0 0x7f5e3c0e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3c046ce0 con 0x7f5e3c0dda40 2024-04-25T11:42:42.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3c0dde50 con 0x7f5e3c0064f0 2024-04-25T11:42:42.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.215+0000 7f5e49dac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5e3c0e7310 con 0x7f5e3c0468d0 2024-04-25T11:42:42.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e43fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5e3c0dda40 0x7f5e3c0ee050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:50926/0 (socket says 172.21.15.136:50926) 2024-04-25T11:42:42.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e43fff640 1 -- 172.21.15.136:0/2757478549 learned_addr learned my addr 172.21.15.136:0/2757478549 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:42.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971064274 0 0) 0x7f5e44050cc0 con 0x7f5e3c0dda40 2024-04-25T11:42:42.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e42ffd640 1 --2- 172.21.15.136:0/2757478549 >> v2:172.21.15.157:3300/0 conn(0x7f5e3c0468d0 0x7f5e3c0e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e34003530 con 0x7f5e3c0dda40 2024-04-25T11:42:42.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4075987091 0 0) 0x7f5e34003530 con 0x7f5e3c0dda40 2024-04-25T11:42:42.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 >> v2:172.21.15.157:3300/0 conn(0x7f5e3c0468d0 msgr2=0x7f5e3c0e77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:42.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 --2- 172.21.15.136:0/2757478549 >> v2:172.21.15.157:3300/0 conn(0x7f5e3c0468d0 0x7f5e3c0e77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 >> v1:172.21.15.157:6789/0 conn(0x7f5e3c0064f0 legacy=0x7f5e3c0457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c047970 con 0x7f5e3c0dda40 2024-04-25T11:42:42.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e42ffd640 1 --2- 172.21.15.136:0/2757478549 >> v2:172.21.15.157:3300/0 conn(0x7f5e3c0468d0 0x7f5e3c0e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:42:42.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f5e4406a2e0 con 0x7f5e3c0dda40 2024-04-25T11:42:42.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 >> v1:172.21.15.136:6789/0 conn(0x7f5e3c0dda40 legacy=0x7f5e3c0ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 0x7f5e34003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e437fe640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 0x7f5e34003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.210 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.219+0000 7f5e427fc640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e34004810 0x7f5e34004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e427fc640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e34005160 0x7f5e3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e42ffd640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e34004810 0x7f5e34004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c0e7310 con 0x7f5e340039c0 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e43fff640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e34005160 0x7f5e3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3400bf90 con 0x7f5e34004810 2024-04-25T11:42:42.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3400c190 con 0x7f5e34005160 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e437fe640 1 -- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e34005160 msgr2=0x7f5e3400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e437fe640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e34005160 0x7f5e3400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e437fe640 1 -- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e34004810 msgr2=0x7f5e34004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e437fe640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e34004810 0x7f5e34004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.223+0000 7f5e437fe640 1 -- 172.21.15.136:0/2757478549 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e34003530 con 0x7f5e340039c0 2024-04-25T11:42:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e437fe640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 0x7f5e34003df0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5e38001890 tx=0x7f5e3801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e6e81c4588278f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:42.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e42ffd640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e34004810 0x7f5e34004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:42.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e43fff640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e34005160 0x7f5e3400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:42.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e43fff640 1 -- 172.21.15.136:0/2757478549 reap_dead start 2024-04-25T11:42:42.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e380b5070 con 0x7f5e340039c0 2024-04-25T11:42:42.213 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e427fc640 1 -- 172.21.15.136:0/2757478549 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e380a8850 con 0x7f5e340039c0 2024-04-25T11:42:42.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e49dac640 1 -- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 msgr2=0x7f5e34003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.227+0000 7f5e49dac640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 0x7f5e34003df0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5e38001890 tx=0x7f5e3801ec10 comp rx=0 tx=0).stop 2024-04-25T11:42:42.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.235+0000 7f5e49dac640 1 -- 172.21.15.136:0/2757478549 shutdown_connections 2024-04-25T11:42:42.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.235+0000 7f5e49dac640 1 --2- 172.21.15.136:0/2757478549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e340039c0 0x7f5e34003df0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.235+0000 7f5e49dac640 1 -- 172.21.15.136:0/2757478549 >> 172.21.15.136:0/2757478549 conn(0x7f5e3c050810 msgr2=0x7f5e3c050c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:42.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.239+0000 7f5e49dac640 1 -- 172.21.15.136:0/2757478549 shutdown_connections 2024-04-25T11:42:42.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.239+0000 7f5e49dac640 1 -- 172.21.15.136:0/2757478549 wait complete. 2024-04-25T11:42:42.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.239+0000 7f5e49dac640 1 Processor -- start 2024-04-25T11:42:42.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 -- start start 2024-04-25T11:42:42.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 0x7f5e3c16ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e3c16da90 0x7f5e3c04f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 0x7f5e3c04faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c047500 con 0x7f5e3c16dea0 2024-04-25T11:42:42.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e437fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 0x7f5e3c16ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e437fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 0x7f5e3c16ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:56086/0 (socket says 172.21.15.136:56086) 2024-04-25T11:42:42.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e437fe640 1 -- 172.21.15.136:0/4191516725 learned_addr learned my addr 172.21.15.136:0/4191516725 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:42.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e42ffd640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e3c16da90 0x7f5e3c04f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c047220 con 0x7f5e3c16da90 2024-04-25T11:42:42.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.243+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3c0e6cc0 con 0x7f5e340039c0 2024-04-25T11:42:42.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 0x7f5e3c04faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 -- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 msgr2=0x7f5e3c16ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 0x7f5e3c16ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 -- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e3c16da90 msgr2=0x7f5e3c04f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e3c16da90 0x7f5e3c04f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c16c420 con 0x7f5e3c16dea0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e43fff640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 0x7f5e3c04faa0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5e44064210 tx=0x7f5e4406da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4406a2e0 con 0x7f5e3c16dea0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3c036580 con 0x7f5e3c16dea0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3c0d09b0 con 0x7f5e3c16dea0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e44095220 con 0x7f5e3c16dea0 2024-04-25T11:42:42.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.247+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44095610 con 0x7f5e3c16dea0 2024-04-25T11:42:42.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.251+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5e44097020 con 0x7f5e3c16dea0 2024-04-25T11:42:42.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.251+0000 7f5e2ffff640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 0x7f5e28071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:42.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.251+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f5e4410b990 con 0x7f5e3c16dea0 2024-04-25T11:42:42.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.251+0000 7f5e437fe640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 0x7f5e28071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:42.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.255+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e240052a0 con 0x7f5e3c16dea0 2024-04-25T11:42:42.241 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.255+0000 7f5e437fe640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 0x7f5e28071910 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e3801ec50 tx=0x7f5e380012b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:42.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.267+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5e3c046ce0 con 0x7f5e3c16dea0 2024-04-25T11:42:42.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.759+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e24005590 con 0x7f5e3c16dea0 2024-04-25T11:42:42.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.763+0000 7f5e2ffff640 1 -- 172.21.15.136:0/4191516725 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7f5e440d8060 con 0x7f5e3c16dea0 2024-04-25T11:42:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:42.749 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:42.750 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:42.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 msgr2=0x7f5e28071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 0x7f5e28071910 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e3801ec50 tx=0x7f5e380012b0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 msgr2=0x7f5e3c04faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:42.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 0x7f5e3c04faa0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5e44064210 tx=0x7f5e4406da50 comp rx=0 tx=0).stop 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 shutdown_connections 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c16dea0 0x7f5e3c04faa0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e3c16da90 0x7f5e3c04f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5e2806f400 0x7f5e28071910 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 --2- 172.21.15.136:0/4191516725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e340039c0 0x7f5e3c16ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 >> 172.21.15.136:0/4191516725 conn(0x7f5e3c050810 msgr2=0x7f5e3c04bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:42.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 shutdown_connections 2024-04-25T11:42:42.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:42.803+0000 7f5e49dac640 1 -- 172.21.15.136:0/4191516725 wait complete. 2024-04-25T11:42:42.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:42:44.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:45.249 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.259+0000 7fa51c967640 1 Processor -- start 2024-04-25T11:42:45.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.259+0000 7fa51c967640 1 -- start start 2024-04-25T11:42:45.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.271+0000 7fa51c967640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa510046e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:45.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.271+0000 7fa515d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa5100ddc40 0x7fa510006270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:35322/0 (socket says 172.21.15.136:35322) 2024-04-25T11:42:45.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.271+0000 7fa515d9b640 1 -- 172.21.15.136:0/3217089245 learned_addr learned my addr 172.21.15.136:0/3217089245 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:45.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.271+0000 7fa51659c640 1 --2- 172.21.15.136:0/3217089245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa510046e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:45.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.271+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5100e4450 con 0x7fa5100ddc40 2024-04-25T11:42:45.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.275+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa510046360 con 0x7fa51004c8a0 2024-04-25T11:42:45.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.275+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa5100461c0 con 0x7fa510045360 2024-04-25T11:42:45.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.275+0000 7fa51659c640 1 -- 172.21.15.136:0/3217089245 >> v1:172.21.15.157:6789/0 conn(0x7fa51004c8a0 legacy=0x7fa5100ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:45.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.275+0000 7fa51659c640 1 -- 172.21.15.136:0/3217089245 >> v1:172.21.15.136:6789/0 conn(0x7fa5100ddc40 legacy=0x7fa510006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:45.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.275+0000 7fa51659c640 1 -- 172.21.15.136:0/3217089245 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5100342c0 con 0x7fa510045360 2024-04-25T11:42:45.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.283+0000 7fa51659c640 1 --2- 172.21.15.136:0/3217089245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa510046e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097f0 tx=0x7fa50801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6433ee5789fb5ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:45.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.295+0000 7fa51559a640 1 -- 172.21.15.136:0/3217089245 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080b0070 con 0x7fa510045360 2024-04-25T11:42:45.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.295+0000 7fa51559a640 1 -- 172.21.15.136:0/3217089245 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5080ab600 con 0x7fa510045360 2024-04-25T11:42:45.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.295+0000 7fa51559a640 1 -- 172.21.15.136:0/3217089245 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080ab9d0 con 0x7fa510045360 2024-04-25T11:42:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.303+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 msgr2=0x7fa510046e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.303+0000 7fa51c967640 1 --2- 172.21.15.136:0/3217089245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa510046e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097f0 tx=0x7fa50801c330 comp rx=0 tx=0).stop 2024-04-25T11:42:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.303+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 shutdown_connections 2024-04-25T11:42:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.303+0000 7fa51c967640 1 --2- 172.21.15.136:0/3217089245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa510046e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:45.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.303+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 >> 172.21.15.136:0/3217089245 conn(0x7fa5100503c0 msgr2=0x7fa5100507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:45.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.315+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 shutdown_connections 2024-04-25T11:42:45.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.315+0000 7fa51c967640 1 -- 172.21.15.136:0/3217089245 wait complete. 2024-04-25T11:42:45.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.319+0000 7fa51c967640 1 Processor -- start 2024-04-25T11:42:45.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 -- start start 2024-04-25T11:42:45.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa51004c8a0 0x7fa5100e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5100ddc40 0x7fa51017d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa510046620 con 0x7fa51004c8a0 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa510046990 con 0x7fa5100ddc40 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51c967640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa51004ad40 con 0x7fa510045360 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:42406/0 (socket says 172.21.15.136:42406) 2024-04-25T11:42:45.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 -- 172.21.15.136:0/449176126 learned_addr learned my addr 172.21.15.136:0/449176126 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa515d9b640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa51004c8a0 0x7fa5100e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa516d9d640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5100ddc40 0x7fa51017d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 -- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5100ddc40 msgr2=0x7fa51017d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5100ddc40 0x7fa51017d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 -- 172.21.15.136:0/449176126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa51004c8a0 msgr2=0x7fa5100e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa51004c8a0 0x7fa5100e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.323+0000 7fa51659c640 1 -- 172.21.15.136:0/449176126 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5100e05c0 con 0x7fa510045360 2024-04-25T11:42:45.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa51659c640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097f0 tx=0x7fa50801ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:45.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080b0050 con 0x7fa510045360 2024-04-25T11:42:45.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5080bc070 con 0x7fa510045360 2024-04-25T11:42:45.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080aad60 con 0x7fa510045360 2024-04-25T11:42:45.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5100de330 con 0x7fa510045360 2024-04-25T11:42:45.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.327+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5100df5a0 con 0x7fa510045360 2024-04-25T11:42:45.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.331+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa510005580 con 0x7fa510045360 2024-04-25T11:42:45.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.331+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa5080b7450 con 0x7fa510045360 2024-04-25T11:42:45.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.331+0000 7fa506ffd640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 0x7fa4f8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:45.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.331+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7fa50812a520 con 0x7fa510045360 2024-04-25T11:42:45.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.331+0000 7fa515d9b640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 0x7fa4f8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:45.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.339+0000 7fa515d9b640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 0x7fa4f8071940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa510177bd0 tx=0x7fa50c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:45.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:45.339+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa508015ff0 con 0x7fa510045360 2024-04-25T11:42:46.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.339+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa510033f30 con 0x7fa510045360 2024-04-25T11:42:46.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.339+0000 7fa506ffd640 1 -- 172.21.15.136:0/449176126 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7fa5080f6c00 con 0x7fa510045360 2024-04-25T11:42:46.329 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:46.330 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:46.330 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:46.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.347+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 msgr2=0x7fa4f8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:46.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.347+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 0x7fa4f8071940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa510177bd0 tx=0x7fa50c00b040 comp rx=0 tx=0).stop 2024-04-25T11:42:46.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.347+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 msgr2=0x7fa5100e0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:46.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.347+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5080097f0 tx=0x7fa50801ab70 comp rx=0 tx=0).stop 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 shutdown_connections 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5100ddc40 0x7fa51017d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa51004c8a0 0x7fa5100e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa4f806f430 0x7fa4f8071940 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 --2- 172.21.15.136:0/449176126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa510045360 0x7fa5100e0080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:46.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.383+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 >> 172.21.15.136:0/449176126 conn(0x7fa5100503c0 msgr2=0x7fa5100348d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:46.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.391+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 shutdown_connections 2024-04-25T11:42:46.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:46.391+0000 7fa51c967640 1 -- 172.21.15.136:0/449176126 wait complete. 2024-04-25T11:42:46.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:42:48.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:48.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 Processor -- start 2024-04-25T11:42:48.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 -- start start 2024-04-25T11:42:48.926 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3bcc0d9730 0x7f3bcc046c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bcc0e0300 con 0x7f3bcc0471b0 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bcc04dcd0 con 0x7f3bcc006bb0 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd9c05640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3bcc0083d0 con 0x7f3bcc0d9730 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3bcc0471b0 0x7f3bcc0e9d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:35328/0 (socket says 172.21.15.136:35328) 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd3fff640 1 -- 172.21.15.136:0/1137170419 learned_addr learned my addr 172.21.15.136:0/1137170419 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd2ffd640 1 --2- 172.21.15.136:0/1137170419 >> v2:172.21.15.157:3300/0 conn(0x7f3bcc0d9730 0x7f3bcc046c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.938+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245863040 0 0) 0x7f3bcc0e0300 con 0x7f3bcc0471b0 2024-04-25T11:42:48.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bc4003530 con 0x7f3bcc0471b0 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1095264354 0 0) 0x7f3bcc04dcd0 con 0x7f3bcc006bb0 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bcc0e0300 con 0x7f3bcc006bb0 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1137170419 >> v1:172.21.15.157:6789/0 conn(0x7f3bcc006bb0 legacy=0x7f3bcc04be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1137170419 >> v1:172.21.15.136:6789/0 conn(0x7f3bcc0471b0 legacy=0x7f3bcc0e9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.942+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1137170419 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc007330 con 0x7f3bcc0d9730 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.946+0000 7f3bd2ffd640 1 --2- 172.21.15.136:0/1137170419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc046c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00097c0 tx=0x7f3bc001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bffb667b242b2e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.946+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00b5070 con 0x7f3bcc0d9730 2024-04-25T11:42:48.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc00a8950 con 0x7f3bcc0d9730 2024-04-25T11:42:48.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd27fc640 1 -- 172.21.15.136:0/1137170419 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00a8d20 con 0x7f3bcc0d9730 2024-04-25T11:42:48.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1137170419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 msgr2=0x7f3bcc046c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:48.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1137170419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc046c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00097c0 tx=0x7f3bc001ec50 comp rx=0 tx=0).stop 2024-04-25T11:42:48.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1137170419 shutdown_connections 2024-04-25T11:42:48.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1137170419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc046c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:48.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.958+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1137170419 >> 172.21.15.136:0/1137170419 conn(0x7f3bcc0514d0 msgr2=0x7f3bcc0518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:48.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.970+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1137170419 shutdown_connections 2024-04-25T11:42:48.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.974+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1137170419 wait complete. 2024-04-25T11:42:48.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.974+0000 7f3bd9c05640 1 Processor -- start 2024-04-25T11:42:48.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.974+0000 7f3bd9c05640 1 -- start start 2024-04-25T11:42:48.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 0x7f3bcc179840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:48.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:48.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 0x7f3bcc179840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:48.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 0x7f3bcc179840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:42420/0 (socket says 172.21.15.136:42420) 2024-04-25T11:42:48.965 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56502/0 (socket says 172.21.15.136:56502) 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc1760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd37fe640 1 -- 172.21.15.136:0/1229090732 learned_addr learned my addr 172.21.15.136:0/1229090732 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc0078f0 con 0x7f3bcc0471b0 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc007f40 con 0x7f3bcc0d9730 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3bcc007bd0 con 0x7f3bcc006bb0 2024-04-25T11:42:48.966 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd3fff640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc1760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 msgr2=0x7f3bcc179840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 0x7f3bcc179840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 msgr2=0x7f3bcc1760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc1760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc179d80 con 0x7f3bcc0471b0 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd3fff640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc1760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:48.967 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:48.978+0000 7f3bd2ffd640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0e04f0 tx=0x7f3bc001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:49.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.014+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00b5040 con 0x7f3bcc0471b0 2024-04-25T11:42:49.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.014+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc0dda90 con 0x7f3bcc0471b0 2024-04-25T11:42:49.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.014+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc0de030 con 0x7f3bcc0471b0 2024-04-25T11:42:49.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.018+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc005e10 con 0x7f3bcc0471b0 2024-04-25T11:42:49.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.022+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc00bf070 con 0x7f3bcc0471b0 2024-04-25T11:42:49.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.034+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00b0a30 con 0x7f3bcc0471b0 2024-04-25T11:42:49.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.034+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3bc00b0c50 con 0x7f3bcc0471b0 2024-04-25T11:42:49.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.034+0000 7f3bbffff640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 0x7f3bb8071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:49.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.038+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7f3bc012cb20 con 0x7f3bcc0471b0 2024-04-25T11:42:49.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.038+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3bc00187d0 con 0x7f3bcc0471b0 2024-04-25T11:42:49.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.038+0000 7f3bd37fe640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 0x7f3bb8071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:49.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:49.038+0000 7f3bd37fe640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 0x7f3bb8071a40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406c040 tx=0x7f3bd406f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:50.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.158+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bcc0dddc0 con 0x7f3bcc0471b0 2024-04-25T11:42:50.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.162+0000 7f3bbffff640 1 -- 172.21.15.136:0/1229090732 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7f3bc00f8b90 con 0x7f3bcc0471b0 2024-04-25T11:42:50.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:50.150 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:50.150 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:50.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 msgr2=0x7f3bb8071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:50.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 0x7f3bb8071a40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406c040 tx=0x7f3bd406f040 comp rx=0 tx=0).stop 2024-04-25T11:42:50.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 msgr2=0x7f3bcc1689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:50.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0e04f0 tx=0x7f3bc001b9f0 comp rx=0 tx=0).stop 2024-04-25T11:42:50.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 shutdown_connections 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bcc0d9730 0x7f3bcc1760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bcc0471b0 0x7f3bcc1689b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3bb806f530 0x7f3bb8071a40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 --2- 172.21.15.136:0/1229090732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bcc006bb0 0x7f3bcc179840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 >> 172.21.15.136:0/1229090732 conn(0x7f3bcc0514d0 msgr2=0x7f3bcc0d7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 shutdown_connections 2024-04-25T11:42:50.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:50.174+0000 7f3bd9c05640 1 -- 172.21.15.136:0/1229090732 wait complete. 2024-04-25T11:42:50.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:42:52.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:52.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.650+0000 7efff48fe640 1 Processor -- start 2024-04-25T11:42:52.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.650+0000 7efff48fe640 1 -- start start 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7efff48fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efff00b5810 0x7efff00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7efff48fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 0x7efff0166910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7efff48fe640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff00b4480 con 0x7efff015c070 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7efff48fe640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7efff00b4a90 con 0x7efff00b5810 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7efff48fe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efff015c990 con 0x7efff015c480 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7effee59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efff00b5810 0x7efff00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7effee59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efff00b5810 0x7efff00b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56506/0 (socket says 172.21.15.136:56506) 2024-04-25T11:42:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7effee59c640 1 -- 172.21.15.136:0/3583748026 learned_addr learned my addr 172.21.15.136:0/3583748026 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:52.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.654+0000 7effeed9d640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 0x7efff0166910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:52.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.658+0000 7effee59c640 1 -- 172.21.15.136:0/3583748026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 msgr2=0x7efff0166910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:42:52.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.658+0000 7effee59c640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 0x7efff0166910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:52.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.658+0000 7effee59c640 1 -- 172.21.15.136:0/3583748026 >> v1:172.21.15.157:6789/0 conn(0x7efff015c070 legacy=0x7efff006d510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:42:52.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.658+0000 7effee59c640 1 -- 172.21.15.136:0/3583748026 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff015cc70 con 0x7efff00b5810 2024-04-25T11:42:52.643 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.658+0000 7effeed9d640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 0x7efff0166910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:42:52.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.662+0000 7effee59c640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff00b5810 0x7efff00b4f20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ad0 tx=0x7effe401ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bf69ea468b6c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:52.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.662+0000 7effed59a640 1 -- 172.21.15.136:0/3583748026 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe40b5070 con 0x7efff00b5810 2024-04-25T11:42:52.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.662+0000 7effed59a640 1 -- 172.21.15.136:0/3583748026 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe40a8c30 con 0x7efff00b5810 2024-04-25T11:42:52.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 -- 172.21.15.136:0/3583748026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff00b5810 msgr2=0x7efff00b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:52.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff00b5810 0x7efff00b4f20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ad0 tx=0x7effe401ef70 comp rx=0 tx=0).stop 2024-04-25T11:42:52.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 -- 172.21.15.136:0/3583748026 shutdown_connections 2024-04-25T11:42:52.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff015c480 0x7efff0166910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:52.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 --2- 172.21.15.136:0/3583748026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff00b5810 0x7efff00b4f20 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:52.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.666+0000 7efff48fe640 1 -- 172.21.15.136:0/3583748026 >> 172.21.15.136:0/3583748026 conn(0x7efff00743a0 msgr2=0x7efff00d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:52.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.678+0000 7efff48fe640 1 -- 172.21.15.136:0/3583748026 shutdown_connections 2024-04-25T11:42:52.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.682+0000 7efff48fe640 1 -- 172.21.15.136:0/3583748026 wait complete. 2024-04-25T11:42:52.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.690+0000 7efff48fe640 1 Processor -- start 2024-04-25T11:42:52.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.690+0000 7efff48fe640 1 -- start start 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 0x7efff01e53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efff015c480 0x7efff01e5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efff006e030 con 0x7efff015c070 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efff015cf50 con 0x7efff015c480 2024-04-25T11:42:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7efff48fe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efff00b5460 con 0x7efff00b5810 2024-04-25T11:42:52.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.694+0000 7effedd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 0x7efff01e53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:52.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.698+0000 7effee59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:52.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.698+0000 7effee59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:42436/0 (socket says 172.21.15.136:42436) 2024-04-25T11:42:52.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.698+0000 7effee59c640 1 -- 172.21.15.136:0/2830195290 learned_addr learned my addr 172.21.15.136:0/2830195290 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:52.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 -- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efff015c480 msgr2=0x7efff01e5920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:42:52.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efff015c480 0x7efff01e5920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:52.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 -- 172.21.15.136:0/2830195290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 msgr2=0x7efff01e53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:52.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 0x7efff01e53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 -- 172.21.15.136:0/2830195290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff0157c10 con 0x7efff00b5810 2024-04-25T11:42:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effedd9b640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 0x7efff01e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:42:52.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.706+0000 7effee59c640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ad0 tx=0x7effe401bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:52.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.710+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe40b5040 con 0x7efff00b5810 2024-04-25T11:42:52.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.710+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff01e7690 con 0x7efff00b5810 2024-04-25T11:42:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.710+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efff01e7950 con 0x7efff00b5810 2024-04-25T11:42:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.710+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe40c0070 con 0x7efff00b5810 2024-04-25T11:42:52.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.710+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe40b0880 con 0x7efff00b5810 2024-04-25T11:42:52.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.714+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7effe40b0a20 con 0x7efff00b5810 2024-04-25T11:42:52.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.714+0000 7effdeffd640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 0x7effd0071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:52.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.714+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 7767+0+0 (secure 0 0 0) 0x7effe412d230 con 0x7efff00b5810 2024-04-25T11:42:52.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.714+0000 7effedd9b640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 0x7effd0071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:52.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.714+0000 7effedd9b640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 0x7effd0071960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efff00c8a80 tx=0x7effe80084d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:52.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.762+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd40052a0 con 0x7efff00b5810 2024-04-25T11:42:52.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:52.778+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7effe4018ad0 con 0x7efff00b5810 2024-04-25T11:42:53.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.666+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effd4005860 con 0x7efff00b5810 2024-04-25T11:42:53.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.670+0000 7effdeffd640 1 -- 172.21.15.136:0/2830195290 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6934 (secure 0 0 0) 0x7effe40f9a80 con 0x7efff00b5810 2024-04-25T11:42:53.656 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:53.656 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:23.439223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5449,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"f","rank":1,"incarnation":56,"state":"up:active","state_seq":15,"addr":"172.21.15.157:6839/3020784477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3020784477},{"type":"v1","addr":"172.21.15.157:6839","nonce":3020784477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5517,5839,5449]},"id":1}]} 2024-04-25T11:42:53.656 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 91 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.678+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 msgr2=0x7effd0071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.678+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 0x7effd0071960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efff00c8a80 tx=0x7effe80084d0 comp rx=0 tx=0).stop 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.678+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 msgr2=0x7efff01576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.678+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effe4009ad0 tx=0x7effe401bd10 comp rx=0 tx=0).stop 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.678+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 shutdown_connections 2024-04-25T11:42:53.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efff015c480 0x7efff01e5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efff015c070 0x7efff01e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7effd006f450 0x7effd0071960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 --2- 172.21.15.136:0/2830195290 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efff00b5810 0x7efff01576d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 >> 172.21.15.136:0/2830195290 conn(0x7efff00743a0 msgr2=0x7efff0155c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:53.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 shutdown_connections 2024-04-25T11:42:53.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:53.682+0000 7efff48fe640 1 -- 172.21.15.136:0/2830195290 wait complete. 2024-04-25T11:42:53.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-25T11:42:55.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:56.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.254+0000 7f29304b5640 1 Processor -- start 2024-04-25T11:42:56.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.258+0000 7f29304b5640 1 -- start start 2024-04-25T11:42:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.262+0000 7f29304b5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2928143170 0x7f292815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.262+0000 7f292e251640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2928143170 0x7f292815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.262+0000 7f292e251640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2928143170 0x7f292815d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33284/0 (socket says 172.21.15.136:33284) 2024-04-25T11:42:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.262+0000 7f292da50640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2928070870 0x7f29280c6960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41228/0 (socket says 172.21.15.136:41228) 2024-04-25T11:42:56.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.262+0000 7f292e251640 1 -- 172.21.15.136:0/2717741636 learned_addr learned my addr 172.21.15.136:0/2717741636 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2928143580 con 0x7f2928070870 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29280c6760 con 0x7f2928070c80 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f292815dd20 con 0x7f2928143170 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f292e251640 1 -- 172.21.15.136:0/2717741636 >> v1:172.21.15.157:6789/0 conn(0x7f2928070c80 legacy=0x7f2928162780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f292e251640 1 -- 172.21.15.136:0/2717741636 >> v1:172.21.15.136:6789/0 conn(0x7f2928070870 legacy=0x7f29280c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:56.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.266+0000 7f292e251640 1 -- 172.21.15.136:0/2717741636 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292815e000 con 0x7f2928143170 2024-04-25T11:42:56.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f292e251640 1 --2- 172.21.15.136:0/2717741636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928143170 0x7f292815d400 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ad0 tx=0x7f292402d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47e1f6716897eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:56.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f292d24f640 1 -- 172.21.15.136:0/2717741636 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240ba650 con 0x7f2928143170 2024-04-25T11:42:56.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f292d24f640 1 -- 172.21.15.136:0/2717741636 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29240ba7f0 con 0x7f2928143170 2024-04-25T11:42:56.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f292d24f640 1 -- 172.21.15.136:0/2717741636 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240babc0 con 0x7f2928143170 2024-04-25T11:42:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928143170 msgr2=0x7f292815d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:56.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.270+0000 7f29304b5640 1 --2- 172.21.15.136:0/2717741636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928143170 0x7f292815d400 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2924009ad0 tx=0x7f292402d5a0 comp rx=0 tx=0).stop 2024-04-25T11:42:56.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.274+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 shutdown_connections 2024-04-25T11:42:56.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.274+0000 7f29304b5640 1 --2- 172.21.15.136:0/2717741636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928143170 0x7f292815d400 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:56.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.274+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 >> 172.21.15.136:0/2717741636 conn(0x7f29280c9d50 msgr2=0x7f29280ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:56.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.286+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 shutdown_connections 2024-04-25T11:42:56.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.286+0000 7f29304b5640 1 -- 172.21.15.136:0/2717741636 wait complete. 2024-04-25T11:42:56.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.286+0000 7f29304b5640 1 Processor -- start 2024-04-25T11:42:56.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 -- start start 2024-04-25T11:42:56.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 0x7f292813b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:56.277 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f292e251640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 0x7f292813b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f292e251640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 0x7f292813b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:33894/0 (socket says 172.21.15.136:33894) 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f292da50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f292da50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33286/0 (socket says 172.21.15.136:33286) 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f292e251640 1 -- 172.21.15.136:0/3055765206 learned_addr learned my addr 172.21.15.136:0/3055765206 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2928143170 0x7f29281ee860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:56.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2928070350 con 0x7f2928070c80 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2928070070 con 0x7f2928143170 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.290+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f292815e2e0 con 0x7f2928070870 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 -- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 msgr2=0x7f292813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 0x7f292813b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 -- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2928143170 msgr2=0x7f29281ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2928143170 0x7f29281ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:56.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292813bce0 con 0x7f2928070c80 2024-04-25T11:42:56.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f292da50640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2928071ef0 tx=0x7f291800ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:56.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291801b070 con 0x7f2928070c80 2024-04-25T11:42:56.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291800a4e0 con 0x7f2928070c80 2024-04-25T11:42:56.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.294+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2928153fb0 con 0x7f2928070c80 2024-04-25T11:42:56.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.298+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292813c420 con 0x7f2928070c80 2024-04-25T11:42:56.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.298+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918007a30 con 0x7f2928070c80 2024-04-25T11:42:56.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.298+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f291801e070 con 0x7f2928070c80 2024-04-25T11:42:56.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.298+0000 7f291effd640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 0x7f290c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:56.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.298+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f29180924a0 con 0x7f2928070c80 2024-04-25T11:42:56.291 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.306+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29100052a0 con 0x7f2928070c80 2024-04-25T11:42:56.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.306+0000 7f292e251640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 0x7f290c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:56.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.310+0000 7f292e251640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 0x7f290c071a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f292402bc50 tx=0x7f2924008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:56.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:56.314+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2928143580 con 0x7f2928070c80 2024-04-25T11:42:57.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:42:57.161 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:42:57.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.206+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2910005860 con 0x7f2928070c80 2024-04-25T11:42:57.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.210+0000 7f291effd640 1 -- 172.21.15.136:0/3055765206 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6101 (secure 0 0 0) 0x7f291805e240 con 0x7f2928070c80 2024-04-25T11:42:57.199 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:57.199 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:54.061723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,6246]},"id":1}]} 2024-04-25T11:42:57.199 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 92 2024-04-25T11:42:57.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 msgr2=0x7f290c071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 0x7f290c071a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f292402bc50 tx=0x7f2924008040 comp rx=0 tx=0).stop 2024-04-25T11:42:57.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 msgr2=0x7f2928153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2928071ef0 tx=0x7f291800ff50 comp rx=0 tx=0).stop 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 shutdown_connections 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2928143170 0x7f29281ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2928070c80 0x7f2928153a70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f290c06f500 0x7f290c071a10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 --2- 172.21.15.136:0/3055765206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2928070870 0x7f292813b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 >> 172.21.15.136:0/3055765206 conn(0x7f29280c9d50 msgr2=0x7f2928141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 shutdown_connections 2024-04-25T11:42:57.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.214+0000 7f29304b5640 1 -- 172.21.15.136:0/3055765206 wait complete. 2024-04-25T11:42:57.357 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-25T11:42:57.358 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:42:57.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:42:57.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.742+0000 7fe263fff640 1 Processor -- start 2024-04-25T11:42:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.742+0000 7fe263fff640 1 -- start start 2024-04-25T11:42:57.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.746+0000 7fe263fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe25c047b20 0x7fe25c0d3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:57.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.746+0000 7fe2627fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe25c047460 0x7fe25c0ddd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41240/0 (socket says 172.21.15.136:41240) 2024-04-25T11:42:57.742 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.754+0000 7fe2627fc640 1 -- 172.21.15.136:0/2192875761 learned_addr learned my addr 172.21.15.136:0/2192875761 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:42:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.762+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe25c0e1800 con 0x7fe25c047460 2024-04-25T11:42:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.762+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe25c04d400 con 0x7fe25c04eff0 2024-04-25T11:42:57.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.762+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe25c0356a0 con 0x7fe25c047b20 2024-04-25T11:42:57.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.766+0000 7fe262ffd640 1 --2- 172.21.15.136:0/2192875761 >> v2:172.21.15.157:3300/0 conn(0x7fe25c047b20 0x7fe25c0d3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:57.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.766+0000 7fe262ffd640 1 -- 172.21.15.136:0/2192875761 >> v1:172.21.15.157:6789/0 conn(0x7fe25c04eff0 legacy=0x7fe25c0481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.766+0000 7fe262ffd640 1 -- 172.21.15.136:0/2192875761 >> v1:172.21.15.136:6789/0 conn(0x7fe25c047460 legacy=0x7fe25c0ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.766+0000 7fe262ffd640 1 -- 172.21.15.136:0/2192875761 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25c04d6e0 con 0x7fe25c047b20 2024-04-25T11:42:57.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.770+0000 7fe262ffd640 1 --2- 172.21.15.136:0/2192875761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c047b20 0x7fe25c0d3ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe254009850 tx=0x7fe254020650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f41e0661efc5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:57.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.778+0000 7fe261ffb640 1 -- 172.21.15.136:0/2192875761 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2540ad680 con 0x7fe25c047b20 2024-04-25T11:42:57.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.778+0000 7fe261ffb640 1 -- 172.21.15.136:0/2192875761 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2540ad820 con 0x7fe25c047b20 2024-04-25T11:42:57.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.778+0000 7fe261ffb640 1 -- 172.21.15.136:0/2192875761 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2540adbf0 con 0x7fe25c047b20 2024-04-25T11:42:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.778+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c047b20 msgr2=0x7fe25c0d3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.778+0000 7fe263fff640 1 --2- 172.21.15.136:0/2192875761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c047b20 0x7fe25c0d3ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe254009850 tx=0x7fe254020650 comp rx=0 tx=0).stop 2024-04-25T11:42:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.782+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 shutdown_connections 2024-04-25T11:42:57.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.782+0000 7fe263fff640 1 --2- 172.21.15.136:0/2192875761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c047b20 0x7fe25c0d3ff0 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe254009850 tx=0x7fe254020650 comp rx=0 tx=0).stop 2024-04-25T11:42:57.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.782+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 >> 172.21.15.136:0/2192875761 conn(0x7fe25c009fe0 msgr2=0x7fe25c00a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:57.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.782+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 shutdown_connections 2024-04-25T11:42:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.782+0000 7fe263fff640 1 -- 172.21.15.136:0/2192875761 wait complete. 2024-04-25T11:42:57.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.786+0000 7fe263fff640 1 Processor -- start 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 -- start start 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe25c047460 0x7fe25c0095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 0x7fe25c0cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 0x7fe25c0cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c04dbb0 con 0x7fe25c047460 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c0dd800 con 0x7fe25c171c40 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe263fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe25c04ec20 con 0x7fe25c04eff0 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 0x7fe25c0cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 0x7fe25c0cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:45368/0 (socket says 172.21.15.136:45368) 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.790+0000 7fe2637fe640 1 -- 172.21.15.136:0/1363322327 learned_addr learned my addr 172.21.15.136:0/1363322327 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:42:57.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 0x7fe25c0cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 -- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 msgr2=0x7fe25c0cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 0x7fe25c0cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 -- 172.21.15.136:0/1363322327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe25c047460 msgr2=0x7fe25c0095d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:42:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe25c047460 0x7fe25c0095d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 -- 172.21.15.136:0/1363322327 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25c00a870 con 0x7fe25c04eff0 2024-04-25T11:42:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.794+0000 7fe2627fc640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 0x7fe25c0cd0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe258009d50 tx=0x7fe25800c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258019070 con 0x7fe25c04eff0 2024-04-25T11:42:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25c00a590 con 0x7fe25c04eff0 2024-04-25T11:42:57.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25c04c480 con 0x7fe25c04eff0 2024-04-25T11:42:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2580078f0 con 0x7fe25c04eff0 2024-04-25T11:42:57.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258007ce0 con 0x7fe25c04eff0 2024-04-25T11:42:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe25801c070 con 0x7fe25c04eff0 2024-04-25T11:42:57.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.802+0000 7fe24b7fe640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 0x7fe250071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:42:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.806+0000 7fe262ffd640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 0x7fe250071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:42:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.806+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fe2580915d0 con 0x7fe25c04eff0 2024-04-25T11:42:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.806+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe25c0059d0 con 0x7fe25c04eff0 2024-04-25T11:42:57.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.806+0000 7fe262ffd640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 0x7fe250071940 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe2540b2000 tx=0x7fe254008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:42:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:57.818+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe258009b70 con 0x7fe25c04eff0 2024-04-25T11:42:58.657 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:42:58.657 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:54.061723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,6246]},"id":1}]} 2024-04-25T11:42:58.657 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 92 2024-04-25T11:42:58.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.670+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe25c0dd050 con 0x7fe25c04eff0 2024-04-25T11:42:58.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.670+0000 7fe24b7fe640 1 -- 172.21.15.136:0/1363322327 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6101 (secure 0 0 0) 0x7fe25805d690 con 0x7fe25c04eff0 2024-04-25T11:42:58.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 msgr2=0x7fe250071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 0x7fe250071940 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe2540b2000 tx=0x7fe254008040 comp rx=0 tx=0).stop 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 msgr2=0x7fe25c0cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 0x7fe25c0cd0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe258009d50 tx=0x7fe25800c430 comp rx=0 tx=0).stop 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 shutdown_connections 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe25c171c40 0x7fe25c0cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe25006f430 0x7fe250071940 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe25c04eff0 0x7fe25c0cd0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.674+0000 7fe263fff640 1 --2- 172.21.15.136:0/1363322327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe25c047460 0x7fe25c0095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:42:58.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.678+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 >> 172.21.15.136:0/1363322327 conn(0x7fe25c009fe0 msgr2=0x7fe25c166730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:42:58.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.678+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 shutdown_connections 2024-04-25T11:42:58.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:42:58.678+0000 7fe263fff640 1 -- 172.21.15.136:0/1363322327 wait complete. 2024-04-25T11:42:58.714 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5977": { "addr": "172.21.15.157:6837/3388366693", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3388366693, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3388366693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5977, "incarnation": 92, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:42:54.061723+0000", "qdb_cluster": [ 5517, 5839, 5911, 6246 ], "qdb_leader": 5911, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5977, "mds_2": 5517, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:43:00.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:01.034 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.046+0000 7f9526531640 1 Processor -- start 2024-04-25T11:43:01.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.046+0000 7f9526531640 1 -- start start 2024-04-25T11:43:01.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95200ca2a0 0x7f9520152190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f95200af640 0x7f95200afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f952006ea50 con 0x7f95200ca2a0 2024-04-25T11:43:01.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f95200c3640 con 0x7f95200af640 2024-04-25T11:43:01.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f9526531640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95200c3920 con 0x7f95200c3010 2024-04-25T11:43:01.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f95177fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:01.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f95177fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:33906/0 (socket says 172.21.15.136:33906) 2024-04-25T11:43:01.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f95177fe640 1 -- 172.21.15.136:0/3301783565 learned_addr learned my addr 172.21.15.136:0/3301783565 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:01.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f95177fe640 1 -- 172.21.15.136:0/3301783565 >> v2:172.21.15.157:3300/0 conn(0x7f95200af640 msgr2=0x7f95200afa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:43:01.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.050+0000 7f95177fe640 1 --2- 172.21.15.136:0/3301783565 >> v2:172.21.15.157:3300/0 conn(0x7f95200af640 0x7f95200afa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.054+0000 7f95177fe640 1 -- 172.21.15.136:0/3301783565 >> v2:172.21.15.136:3300/0 conn(0x7f95200ca2a0 msgr2=0x7f9520152190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:43:01.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.054+0000 7f95177fe640 1 --2- 172.21.15.136:0/3301783565 >> v2:172.21.15.136:3300/0 conn(0x7f95200ca2a0 0x7f9520152190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.054+0000 7f95177fe640 1 -- 172.21.15.136:0/3301783565 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95200c3c00 con 0x7f95200c3010 2024-04-25T11:43:01.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.054+0000 7f95177fe640 1 --2- 172.21.15.136:0/3301783565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95200719d0 tx=0x7f951001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e0e7c22d9b68e12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:01.044 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.058+0000 7f951f7fe640 1 -- 172.21.15.136:0/3301783565 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100b0070 con 0x7f95200c3010 2024-04-25T11:43:01.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.062+0000 7f951f7fe640 1 -- 172.21.15.136:0/3301783565 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95100ab600 con 0x7f95200c3010 2024-04-25T11:43:01.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.062+0000 7f951f7fe640 1 -- 172.21.15.136:0/3301783565 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100ab9d0 con 0x7f95200c3010 2024-04-25T11:43:01.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.062+0000 7f9526531640 1 -- 172.21.15.136:0/3301783565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 msgr2=0x7f952006e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:01.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.062+0000 7f9526531640 1 --2- 172.21.15.136:0/3301783565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95200719d0 tx=0x7f951001c560 comp rx=0 tx=0).stop 2024-04-25T11:43:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 -- 172.21.15.136:0/3301783565 shutdown_connections 2024-04-25T11:43:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 --2- 172.21.15.136:0/3301783565 >> v2:172.21.15.157:3300/0 conn(0x7f95200af640 0x7f95200afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 --2- 172.21.15.136:0/3301783565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200c3010 0x7f952006e510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 --2- 172.21.15.136:0/3301783565 >> v2:172.21.15.136:3300/0 conn(0x7f95200ca2a0 0x7f9520152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 -- 172.21.15.136:0/3301783565 >> 172.21.15.136:0/3301783565 conn(0x7f95200cdae0 msgr2=0x7f95200c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:01.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.070+0000 7f9526531640 1 -- 172.21.15.136:0/3301783565 shutdown_connections 2024-04-25T11:43:01.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.078+0000 7f9526531640 1 -- 172.21.15.136:0/3301783565 wait complete. 2024-04-25T11:43:01.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.082+0000 7f9526531640 1 Processor -- start 2024-04-25T11:43:01.069 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.082+0000 7f9526531640 1 -- start start 2024-04-25T11:43:01.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f9526531640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f9526531640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f951ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:01.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f951ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:33908/0 (socket says 172.21.15.136:33908) 2024-04-25T11:43:01.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f95177fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:01.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.086+0000 7f95177fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33296/0 (socket says 172.21.15.136:33296) 2024-04-25T11:43:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f9526531640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 0x7f95201514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f9526531640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f952015bfd0 con 0x7f95200c3010 2024-04-25T11:43:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f9526531640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f952015be50 con 0x7f95200ca2a0 2024-04-25T11:43:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f9526531640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f952015c150 con 0x7f95200af640 2024-04-25T11:43:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f951ffff640 1 -- 172.21.15.136:0/2444763883 learned_addr learned my addr 172.21.15.136:0/2444763883 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:01.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.098+0000 7f9524ace640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 0x7f95201514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 -- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 msgr2=0x7f95201514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 0x7f95201514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 -- 172.21.15.136:0/2444763883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 msgr2=0x7f952015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 -- 172.21.15.136:0/2444763883 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9520150f00 con 0x7f95200af640 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f9524ace640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 0x7f95201514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:43:01.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951ffff640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f950800b450 tx=0x7f950800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508015020 con 0x7f95200af640 2024-04-25T11:43:01.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9520147380 con 0x7f95200af640 2024-04-25T11:43:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f95177fe640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T11:43:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9520147920 con 0x7f95200af640 2024-04-25T11:43:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.102+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95080118b0 con 0x7f95200af640 2024-04-25T11:43:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.106+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508011ca0 con 0x7f95200af640 2024-04-25T11:43:01.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.106+0000 7f95167fc640 1 -- 172.21.15.136:0/2444763883 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95040052a0 con 0x7f95200af640 2024-04-25T11:43:01.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.150+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9508018020 con 0x7f95200af640 2024-04-25T11:43:01.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.154+0000 7f951d7fa640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 0x7f95000719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:01.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.154+0000 7f95177fe640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 0x7f95000719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:01.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.158+0000 7f95177fe640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 0x7f95000719e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f95100ad000 tx=0x7f9510008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:01.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.158+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f950808fc60 con 0x7f95200af640 2024-04-25T11:43:01.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:01.158+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f950801f020 con 0x7f95200af640 2024-04-25T11:43:02.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.210+0000 7f95167fc640 1 -- 172.21.15.136:0/2444763883 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9504005590 con 0x7f95200af640 2024-04-25T11:43:02.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.210+0000 7f951d7fa640 1 -- 172.21.15.136:0/2444763883 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+6103 (secure 0 0 0) 0x7f9520150f00 con 0x7f95200af640 2024-04-25T11:43:02.201 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:02.201 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:42:59.979655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:resolve","state_seq":10,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5839,5517]},"id":1}]} 2024-04-25T11:43:02.201 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 93 2024-04-25T11:43:02.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.234+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 msgr2=0x7f95000719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:02.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.234+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 0x7f95000719e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f95100ad000 tx=0x7f9510008040 comp rx=0 tx=0).stop 2024-04-25T11:43:02.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.234+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 msgr2=0x7f952014f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:02.219 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.234+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f950800b450 tx=0x7f950800b910 comp rx=0 tx=0).stop 2024-04-25T11:43:02.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.242+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 shutdown_connections 2024-04-25T11:43:02.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.242+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95200ca2a0 0x7f95201514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:02.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.242+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95200c3010 0x7f952015a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:02.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.246+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f950006f4d0 0x7f95000719e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:02.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.246+0000 7f9526531640 1 --2- 172.21.15.136:0/2444763883 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95200af640 0x7f952014f9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:02.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.246+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 >> 172.21.15.136:0/2444763883 conn(0x7f95200cdae0 msgr2=0x7f95200c8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:02.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.246+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 shutdown_connections 2024-04-25T11:43:02.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:02.246+0000 7f9526531640 1 -- 172.21.15.136:0/2444763883 wait complete. 2024-04-25T11:43:02.266 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:43:04.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:04.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.678+0000 7fe37c84b640 1 Processor -- start 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.678+0000 7fe37c84b640 1 -- start start 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37c84b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe370045390 0x7fe3701004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37c84b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3700457a0 con 0x7fe370100a30 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37c84b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3701011b0 con 0x7fe37004c900 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37c84b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe370045c50 con 0x7fe370045390 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37659c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fe37004c900 0x7fe370007130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:41800/0 (socket says 172.21.15.136:41800) 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37659c640 1 -- 172.21.15.136:0/777836494 learned_addr learned my addr 172.21.15.136:0/777836494 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe375d9b640 1 --2- 172.21.15.136:0/777836494 >> v2:172.21.15.157:3300/0 conn(0x7fe370045390 0x7fe3701004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.682+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222354222 0 0) 0x7fe3701011b0 con 0x7fe37004c900 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe360003510 con 0x7fe37004c900 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 189123287 0 0) 0x7fe36c001120 con 0x7fe370100a30 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3701011b0 con 0x7fe370100a30 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1239475090 0 0) 0x7fe3701011b0 con 0x7fe370100a30 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 >> v2:172.21.15.157:3300/0 conn(0x7fe370045390 msgr2=0x7fe3701004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 --2- 172.21.15.136:0/777836494 >> v2:172.21.15.157:3300/0 conn(0x7fe370045390 0x7fe3701004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 >> v1:172.21.15.157:6789/0 conn(0x7fe37004c900 legacy=0x7fe370007130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37004bf20 con 0x7fe370100a30 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fe36c001850 con 0x7fe370100a30 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 >> v1:172.21.15.136:6789/0 conn(0x7fe370100a30 legacy=0x7fe370107880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360003ce0 0x7fe3600040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3600046f0 0x7fe360008e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 0x7fe36000b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe36000c120 con 0x7fe360009470 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe36000c2d0 con 0x7fe3600046f0 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe36000c4d0 con 0x7fe360003ce0 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe37659c640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360003ce0 0x7fe3600040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 0x7fe36000b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe375d9b640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3600046f0 0x7fe360008e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 -- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360003ce0 msgr2=0x7fe3600040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360003ce0 0x7fe3600040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 -- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3600046f0 msgr2=0x7fe360008e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3600046f0 0x7fe360008e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe376d9d640 1 -- 172.21.15.136:0/777836494 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37004bf20 con 0x7fe360009470 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.694+0000 7fe375d9b640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3600046f0 0x7fe360008e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe376d9d640 1 -- 172.21.15.136:0/777836494 reap_dead start 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe376d9d640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 0x7fe36000b910 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00c340 tx=0x7fe36c01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7d0ca32da07469 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3780640a0 con 0x7fe360009470 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c0a5a40 con 0x7fe360009470 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37559a640 1 -- 172.21.15.136:0/777836494 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c007e60 con 0x7fe360009470 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 -- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 msgr2=0x7fe36000b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 0x7fe36000b910 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00c340 tx=0x7fe36c01cb40 comp rx=0 tx=0).stop 2024-04-25T11:43:04.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 -- 172.21.15.136:0/777836494 shutdown_connections 2024-04-25T11:43:04.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 --2- 172.21.15.136:0/777836494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe360009470 0x7fe36000b910 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 -- 172.21.15.136:0/777836494 >> 172.21.15.136:0/777836494 conn(0x7fe3700503f0 msgr2=0x7fe370050800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:04.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 -- 172.21.15.136:0/777836494 shutdown_connections 2024-04-25T11:43:04.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.698+0000 7fe37c84b640 1 -- 172.21.15.136:0/777836494 wait complete. 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 Processor -- start 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 -- start start 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360009470 0x7fe3700f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:56252/0 (socket says 172.21.15.136:56252) 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 -- 172.21.15.136:0/2903970167 learned_addr learned my addr 172.21.15.136:0/2903970167 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3700f22e0 0x7fe3700ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe370006c10 con 0x7fe3700f1ed0 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe37004c4e0 con 0x7fe3700f22e0 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe3701013a0 con 0x7fe360009470 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe37659c640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360009470 0x7fe3700f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 -- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360009470 msgr2=0x7fe3700f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360009470 0x7fe3700f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 -- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3700f22e0 msgr2=0x7fe3700ee550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:43:04.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3700f22e0 0x7fe3700ee550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:04.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3700f47b0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.706+0000 7fe375d9b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe378000b70 tx=0x7fe37806aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:04.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.710+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37806dbd0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.710+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe370192ef0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.710+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3701934b0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.704 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.710+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3700055b0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37806dd70 con 0x7fe3700f1ed0 2024-04-25T11:43:04.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3780a8620 con 0x7fe3700f1ed0 2024-04-25T11:43:04.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe3780b1390 con 0x7fe3700f1ed0 2024-04-25T11:43:04.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 0x7fe358071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:04.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fe37811d8e0 con 0x7fe3700f1ed0 2024-04-25T11:43:04.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.722+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe3780ae060 con 0x7fe3700f1ed0 2024-04-25T11:43:04.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.726+0000 7fe37659c640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 0x7fe358071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:04.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:04.730+0000 7fe37659c640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 0x7fe358071a10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe368004d30 tx=0x7fe368007d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:05.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.746+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe37004c6e0 con 0x7fe3700f1ed0 2024-04-25T11:43:05.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.750+0000 7fe366ffd640 1 -- 172.21.15.136:0/2903970167 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+6105 (secure 0 0 0) 0x7fe3780e9f50 con 0x7fe3700f1ed0 2024-04-25T11:43:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:05.746 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:05.301655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,6246]},"id":1}]} 2024-04-25T11:43:05.746 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 94 2024-04-25T11:43:05.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.758+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 msgr2=0x7fe358071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:05.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.758+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 0x7fe358071a10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe368004d30 tx=0x7fe368007d80 comp rx=0 tx=0).stop 2024-04-25T11:43:05.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 msgr2=0x7fe3700ee010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:05.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe378000b70 tx=0x7fe37806aca0 comp rx=0 tx=0).stop 2024-04-25T11:43:05.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 shutdown_connections 2024-04-25T11:43:05.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe3700f22e0 0x7fe3700ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:05.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3700f1ed0 0x7fe3700ee010 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:05.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe35806f500 0x7fe358071a10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:05.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 --2- 172.21.15.136:0/2903970167 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe360009470 0x7fe3700f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:05.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.762+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 >> 172.21.15.136:0/2903970167 conn(0x7fe3700503f0 msgr2=0x7fe3700354d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:05.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.774+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 shutdown_connections 2024-04-25T11:43:05.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:05.778+0000 7fe37c84b640 1 -- 172.21.15.136:0/2903970167 wait complete. 2024-04-25T11:43:05.803 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:43:07.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:08.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.262+0000 7fca409d0640 1 Processor -- start 2024-04-25T11:43:08.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.262+0000 7fca409d0640 1 -- start start 2024-04-25T11:43:08.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.266+0000 7fca409d0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fca3c06f910 0x7fca3c0c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.266+0000 7fca409d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 0x7fca3c16ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.266+0000 7fca409d0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3c160270 con 0x7fca3c0c3a90 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.266+0000 7fca409d0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fca3c0c9820 con 0x7fca3c06f910 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.266+0000 7fca409d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c0c92a0 con 0x7fca3c0c29f0 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca39d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fca3c0c3a90 0x7fca3c073b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:33132/0 (socket says 172.21.15.136:33132) 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca39d9b640 1 -- 172.21.15.136:0/1920018087 learned_addr learned my addr 172.21.15.136:0/1920018087 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/1920018087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 0x7fca3c16ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3959a640 1 -- 172.21.15.136:0/1920018087 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541000528 0 0) 0x7fca3c160270 con 0x7fca3c0c3a90 2024-04-25T11:43:08.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3959a640 1 -- 172.21.15.136:0/1920018087 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca24003530 con 0x7fca3c0c3a90 2024-04-25T11:43:08.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3ad9d640 1 -- 172.21.15.136:0/1920018087 >> v2:172.21.15.136:3300/0 conn(0x7fca3c06f910 msgr2=0x7fca3c0c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:08.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/1920018087 >> v2:172.21.15.136:3300/0 conn(0x7fca3c06f910 0x7fca3c0c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:08.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3ad9d640 1 -- 172.21.15.136:0/1920018087 >> v1:172.21.15.157:6789/0 conn(0x7fca3c0c3a90 legacy=0x7fca3c073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:08.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.270+0000 7fca3ad9d640 1 -- 172.21.15.136:0/1920018087 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c06ef20 con 0x7fca3c0c29f0 2024-04-25T11:43:08.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/1920018087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 0x7fca3c16ab10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fca2c0097c0 tx=0x7fca2c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e335405c6d5769c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:08.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca3959a640 1 -- 172.21.15.136:0/1920018087 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0b0070 con 0x7fca3c0c29f0 2024-04-25T11:43:08.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca3959a640 1 -- 172.21.15.136:0/1920018087 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c0ab600 con 0x7fca3c0c29f0 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 -- 172.21.15.136:0/1920018087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 msgr2=0x7fca3c16ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 --2- 172.21.15.136:0/1920018087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 0x7fca3c16ab10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fca2c0097c0 tx=0x7fca2c01c410 comp rx=0 tx=0).stop 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 -- 172.21.15.136:0/1920018087 shutdown_connections 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 --2- 172.21.15.136:0/1920018087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c0c29f0 0x7fca3c16ab10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 --2- 172.21.15.136:0/1920018087 >> v2:172.21.15.136:3300/0 conn(0x7fca3c06f910 0x7fca3c0c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:08.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.286+0000 7fca409d0640 1 -- 172.21.15.136:0/1920018087 >> 172.21.15.136:0/1920018087 conn(0x7fca3c0cdae0 msgr2=0x7fca3c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:08.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.298+0000 7fca409d0640 1 -- 172.21.15.136:0/1920018087 shutdown_connections 2024-04-25T11:43:08.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.298+0000 7fca409d0640 1 -- 172.21.15.136:0/1920018087 wait complete. 2024-04-25T11:43:08.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.302+0000 7fca409d0640 1 Processor -- start 2024-04-25T11:43:08.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.302+0000 7fca409d0640 1 -- start start 2024-04-25T11:43:08.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c06f910 0x7fca3c143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 0x7fca3c136470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c0743b0 con 0x7fca3c162ba0 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c074230 con 0x7fca3c0c3a90 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c074fd0 con 0x7fca3c06f910 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca39d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca39d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:47242/0 (socket says 172.21.15.136:47242) 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca39d9b640 1 -- 172.21.15.136:0/2010619783 learned_addr learned my addr 172.21.15.136:0/2010619783 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:08.294 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3a59c640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c06f910 0x7fca3c143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:08.298 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 0x7fca3c136470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:08.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 -- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c06f910 msgr2=0x7fca3c143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:08.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c06f910 0x7fca3c143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 -- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 msgr2=0x7fca3c165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 -- 172.21.15.136:0/2010619783 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c137280 con 0x7fca3c162ba0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca39d9b640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca3ad9d640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 0x7fca3c136470 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0ca700 tx=0x7fca2c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0b9020 con 0x7fca3c162ba0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c0aa800 con 0x7fca3c162ba0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0b0040 con 0x7fca3c162ba0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 -- 172.21.15.136:0/2010619783 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c161de0 con 0x7fca3c162ba0 2024-04-25T11:43:08.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.306+0000 7fca409d0640 1 -- 172.21.15.136:0/2010619783 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c1623a0 con 0x7fca3c162ba0 2024-04-25T11:43:08.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.326+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fca2c0bc020 con 0x7fca3c162ba0 2024-04-25T11:43:08.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.326+0000 7fca2affd640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 0x7fca1c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:08.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.326+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fca2c129780 con 0x7fca3c162ba0 2024-04-25T11:43:08.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.326+0000 7fca3a59c640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 0x7fca1c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:08.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.326+0000 7fca3a59c640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 0x7fca1c071940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fca300098f0 tx=0x7fca30006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:08.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.334+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca140052a0 con 0x7fca3c162ba0 2024-04-25T11:43:08.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:08.350+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fca2c016090 con 0x7fca3c162ba0 2024-04-25T11:43:09.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.158+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca14005590 con 0x7fca3c162ba0 2024-04-25T11:43:09.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.162+0000 7fca2affd640 1 -- 172.21.15.136:0/2010619783 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+6105 (secure 0 0 0) 0x7fca2c0f5840 con 0x7fca3c162ba0 2024-04-25T11:43:09.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:09.150 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:05.301655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,6246]},"id":1}]} 2024-04-25T11:43:09.150 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 94 2024-04-25T11:43:09.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.162+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 msgr2=0x7fca1c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:09.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.162+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 0x7fca1c071940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fca300098f0 tx=0x7fca30006f90 comp rx=0 tx=0).stop 2024-04-25T11:43:09.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.166+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 msgr2=0x7fca3c136470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:09.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.166+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 0x7fca3c136470 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0ca700 tx=0x7fca2c0191b0 comp rx=0 tx=0).stop 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 shutdown_connections 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca3c162ba0 0x7fca3c136470 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca3c0c3a90 0x7fca3c165330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fca1c06f430 0x7fca1c071940 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 --2- 172.21.15.136:0/2010619783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca3c06f910 0x7fca3c143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:09.182 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.194+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 >> 172.21.15.136:0/2010619783 conn(0x7fca3c0cdae0 msgr2=0x7fca3c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:09.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.210+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 shutdown_connections 2024-04-25T11:43:09.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:09.210+0000 7fca28ff9640 1 -- 172.21.15.136:0/2010619783 wait complete. 2024-04-25T11:43:09.261 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:43:11.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:11.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 Processor -- start 2024-04-25T11:43:11.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 -- start start 2024-04-25T11:43:11.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f34bc06f910 0x7f34bc0c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc0c3a90 0x7f34bc073b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34bc160270 con 0x7f34bc0c29f0 2024-04-25T11:43:11.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc0c9820 con 0x7f34bc06f910 2024-04-25T11:43:11.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.606+0000 7f34c2db9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc0c92a0 con 0x7f34bc0c3a90 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc0c3a90 0x7f34bc073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc0c3a90 0x7f34bc073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:47258/0 (socket says 172.21.15.136:47258) 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 -- 172.21.15.136:0/58914312 learned_addr learned my addr 172.21.15.136:0/58914312 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34c0b55640 1 --2- 172.21.15.136:0/58914312 >> v2:172.21.15.136:3300/0 conn(0x7f34bc06f910 0x7f34bc0c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 -- 172.21.15.136:0/58914312 >> v2:172.21.15.136:3300/0 conn(0x7f34bc06f910 msgr2=0x7f34bc0c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 --2- 172.21.15.136:0/58914312 >> v2:172.21.15.136:3300/0 conn(0x7f34bc06f910 0x7f34bc0c3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 -- 172.21.15.136:0/58914312 >> v1:172.21.15.157:6789/0 conn(0x7f34bc0c29f0 legacy=0x7f34bc1637e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:11.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.610+0000 7f34bbfff640 1 -- 172.21.15.136:0/58914312 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc06ef20 con 0x7f34bc0c3a90 2024-04-25T11:43:11.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.614+0000 7f34bbfff640 1 --2- 172.21.15.136:0/58914312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c3a90 0x7f34bc073b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0c4460 tx=0x7f34b001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfdf4bf0ba6b18ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.614+0000 7f34bb7fe640 1 -- 172.21.15.136:0/58914312 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00b2070 con 0x7f34bc0c3a90 2024-04-25T11:43:11.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.614+0000 7f34bb7fe640 1 -- 172.21.15.136:0/58914312 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b00ad680 con 0x7f34bc0c3a90 2024-04-25T11:43:11.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.614+0000 7f34bb7fe640 1 -- 172.21.15.136:0/58914312 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00ada30 con 0x7f34bc0c3a90 2024-04-25T11:43:11.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.618+0000 7f34c2db9640 1 -- 172.21.15.136:0/58914312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c3a90 msgr2=0x7f34bc073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.612 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.618+0000 7f34c2db9640 1 --2- 172.21.15.136:0/58914312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c3a90 0x7f34bc073b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0c4460 tx=0x7f34b001c470 comp rx=0 tx=0).stop 2024-04-25T11:43:11.613 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.626+0000 7f34c2db9640 1 -- 172.21.15.136:0/58914312 shutdown_connections 2024-04-25T11:43:11.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.626+0000 7f34c2db9640 1 --2- 172.21.15.136:0/58914312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c3a90 0x7f34bc073b20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.626+0000 7f34c2db9640 1 --2- 172.21.15.136:0/58914312 >> v2:172.21.15.136:3300/0 conn(0x7f34bc06f910 0x7f34bc0c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.614 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.626+0000 7f34c2db9640 1 -- 172.21.15.136:0/58914312 >> 172.21.15.136:0/58914312 conn(0x7f34bc0cdae0 msgr2=0x7f34bc0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:11.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.630+0000 7f34c2db9640 1 -- 172.21.15.136:0/58914312 shutdown_connections 2024-04-25T11:43:11.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.634+0000 7f34c2db9640 1 -- 172.21.15.136:0/58914312 wait complete. 2024-04-25T11:43:11.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.634+0000 7f34c2db9640 1 Processor -- start 2024-04-25T11:43:11.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.638+0000 7f34c2db9640 1 -- start start 2024-04-25T11:43:11.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.638+0000 7f34c2db9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 0x7f34bc2059f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.638+0000 7f34c2db9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34c2db9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0c3a90 0x7f34bc13aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34c2db9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc075150 con 0x7f34bc06f910 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34c2db9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc074fd0 con 0x7f34bc0c29f0 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34c2db9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34bc16bb20 con 0x7f34bc0c3a90 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34c0b55640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 0x7f34bc2059f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34bbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34bbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:47260/0 (socket says 172.21.15.136:47260) 2024-04-25T11:43:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.642+0000 7f34bbfff640 1 -- 172.21.15.136:0/4176970351 learned_addr learned my addr 172.21.15.136:0/4176970351 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 -- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0c3a90 msgr2=0x7f34bc13aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0c3a90 0x7f34bc13aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 -- 172.21.15.136:0/4176970351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 msgr2=0x7f34bc2059f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 0x7f34bc2059f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 -- 172.21.15.136:0/4176970351 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc2048f0 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34bbfff640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f34b00b0040 tx=0x7f34b0019210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00bd070 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34bc06e740 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34bc206710 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b00ada30 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.646+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00b2070 con 0x7f34bc0c29f0 2024-04-25T11:43:11.635 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.650+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f349c0052a0 con 0x7f34bc0c29f0 2024-04-25T11:43:11.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.650+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f34b00c0070 con 0x7f34bc0c29f0 2024-04-25T11:43:11.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.650+0000 7f34b97fa640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 0x7f34a80719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:11.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.654+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f34b0129590 con 0x7f34bc0c29f0 2024-04-25T11:43:11.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.654+0000 7f34c0b55640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 0x7f34bc2059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:11.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.658+0000 7f34c0b55640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 0x7f34a80719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:11.646 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.658+0000 7f34c0b55640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 0x7f34a80719e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007eb0 tx=0x7f34b4007400 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:11.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:11.662+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f34b00160f0 con 0x7f34bc0c29f0 2024-04-25T11:43:12.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.294+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f349c005590 con 0x7f34bc0c29f0 2024-04-25T11:43:12.281 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.294+0000 7f34b97fa640 1 -- 172.21.15.136:0/4176970351 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6102 (secure 0 0 0) 0x7f34b00f5c00 con 0x7f34bc0c29f0 2024-04-25T11:43:12.281 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:12.281 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:10.837009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5839,5517]},"id":1}]} 2024-04-25T11:43:12.281 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 95 2024-04-25T11:43:12.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 msgr2=0x7f34a80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 0x7f34a80719e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f34b4007eb0 tx=0x7f34b4007400 comp rx=0 tx=0).stop 2024-04-25T11:43:12.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 msgr2=0x7f34bc2043b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:12.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f34b00b0040 tx=0x7f34b0019210 comp rx=0 tx=0).stop 2024-04-25T11:43:12.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 shutdown_connections 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f34a806f4d0 0x7f34a80719e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0c3a90 0x7f34bc13aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0c29f0 0x7f34bc2043b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 --2- 172.21.15.136:0/4176970351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34bc06f910 0x7f34bc2059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 >> 172.21.15.136:0/4176970351 conn(0x7f34bc0cdae0 msgr2=0x7f34bc16c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 shutdown_connections 2024-04-25T11:43:12.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:12.298+0000 7f34c2db9640 1 -- 172.21.15.136:0/4176970351 wait complete. 2024-04-25T11:43:12.318 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:43:14.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 Processor -- start 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 -- start start 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7ec15a250 0x7fc7ec15a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ec0729c0 con 0x7fc7ec074100 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ec072c80 con 0x7fc7ec06ff80 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f2c50640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc7ec15aba0 con 0x7fc7ec15a250 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f09ec640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc7ec074100 0x7fc7ec0c7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60046/0 (socket says 172.21.15.136:60046) 2024-04-25T11:43:14.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.746+0000 7fc7f09ec640 1 -- 172.21.15.136:0/2629260257 learned_addr learned my addr 172.21.15.136:0/2629260257 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:43:14.737 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.750+0000 7fc7ebfff640 1 --2- 172.21.15.136:0/2629260257 >> v2:172.21.15.136:3300/0 conn(0x7fc7ec15a250 0x7fc7ec15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:14.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.758+0000 7fc7ebfff640 1 -- 172.21.15.136:0/2629260257 >> v1:172.21.15.157:6789/0 conn(0x7fc7ec06ff80 legacy=0x7fc7ec070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:14.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.758+0000 7fc7ebfff640 1 -- 172.21.15.136:0/2629260257 >> v1:172.21.15.136:6789/0 conn(0x7fc7ec074100 legacy=0x7fc7ec0c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:14.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.758+0000 7fc7ebfff640 1 -- 172.21.15.136:0/2629260257 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec072f60 con 0x7fc7ec15a250 2024-04-25T11:43:14.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.766+0000 7fc7ebfff640 1 --2- 172.21.15.136:0/2629260257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec15a250 0x7fc7ec15a660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00097c0 tx=0x7fc7e001c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d37e6872d5c29d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:14.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.770+0000 7fc7eb7fe640 1 -- 172.21.15.136:0/2629260257 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00b0070 con 0x7fc7ec15a250 2024-04-25T11:43:14.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.770+0000 7fc7eb7fe640 1 -- 172.21.15.136:0/2629260257 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e00ab600 con 0x7fc7ec15a250 2024-04-25T11:43:14.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.770+0000 7fc7f2c50640 1 -- 172.21.15.136:0/2629260257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec15a250 msgr2=0x7fc7ec15a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:14.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.770+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/2629260257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec15a250 0x7fc7ec15a660 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00097c0 tx=0x7fc7e001c270 comp rx=0 tx=0).stop 2024-04-25T11:43:14.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.778+0000 7fc7f2c50640 1 -- 172.21.15.136:0/2629260257 shutdown_connections 2024-04-25T11:43:14.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.778+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/2629260257 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec15a250 0x7fc7ec15a660 secure :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00097c0 tx=0x7fc7e001c270 comp rx=0 tx=0).stop 2024-04-25T11:43:14.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.778+0000 7fc7f2c50640 1 -- 172.21.15.136:0/2629260257 >> 172.21.15.136:0/2629260257 conn(0x7fc7ec0c9d50 msgr2=0x7fc7ec0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:14.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.790+0000 7fc7f2c50640 1 -- 172.21.15.136:0/2629260257 shutdown_connections 2024-04-25T11:43:14.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.802+0000 7fc7f2c50640 1 -- 172.21.15.136:0/2629260257 wait complete. 2024-04-25T11:43:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 Processor -- start 2024-04-25T11:43:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 -- start start 2024-04-25T11:43:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7ec06ff80 0x7fc7ec159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec074100 0x7fc7ec0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec073800 con 0x7fc7ec074100 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ec073520 con 0x7fc7ec06ff80 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f2c50640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ec0712c0 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f11ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f11ed640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44826/0 (socket says 172.21.15.136:44826) 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.806+0000 7fc7f11ed640 1 -- 172.21.15.136:0/196385095 learned_addr learned my addr 172.21.15.136:0/196385095 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 -- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7ec06ff80 msgr2=0x7fc7ec159920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7ec06ff80 0x7fc7ec159920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 -- 172.21.15.136:0/196385095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec074100 msgr2=0x7fc7ec0cc0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec074100 0x7fc7ec0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 -- 172.21.15.136:0/196385095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ec150650 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f11ed640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00c4a0 tx=0x7fc7dc00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:14.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc017070 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ec150930 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ec159400 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7dc005480 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.810+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc003820 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc7dc01a070 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.799 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7e97fa640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 0x7fc7d4071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:14.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7f09ec640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 0x7fc7d4071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:14.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fc7dc08ecb0 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7f09ec640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 0x7fc7d4071ab0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4001610 tx=0x7fc7e4001f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:14.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.814+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ec06a350 con 0x7fc7ec0c65b0 2024-04-25T11:43:14.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:14.826+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc7dc05b540 con 0x7fc7ec0c65b0 2024-04-25T11:43:15.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.554+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7ec0c4250 con 0x7fc7ec0c65b0 2024-04-25T11:43:15.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.558+0000 7fc7e97fa640 1 -- 172.21.15.136:0/196385095 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+6102 (secure 0 0 0) 0x7fc7dc05ae40 con 0x7fc7ec0c65b0 2024-04-25T11:43:15.550 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:15.550 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:10.837009+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5911,5839,5517]},"id":1}]} 2024-04-25T11:43:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 95 2024-04-25T11:43:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 msgr2=0x7fc7d4071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:15.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 0x7fc7d4071ab0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4001610 tx=0x7fc7e4001f10 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 msgr2=0x7fc7ec0cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00c4a0 tx=0x7fc7dc00c960 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 shutdown_connections 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc7d406f5a0 0x7fc7d4071ab0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc7ec0c65b0 0x7fc7ec0cc5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ec074100 0x7fc7ec0cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 --2- 172.21.15.136:0/196385095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc7ec06ff80 0x7fc7ec159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 >> 172.21.15.136:0/196385095 conn(0x7fc7ec0c9d50 msgr2=0x7fc7ec06ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 shutdown_connections 2024-04-25T11:43:15.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:15.566+0000 7fc7f2c50640 1 -- 172.21.15.136:0/196385095 wait complete. 2024-04-25T11:43:15.639 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 95, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5517": { "addr": "172.21.15.136:6837/1740633057", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 1740633057, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 1740633057, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5517, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 27 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5977": { "addr": "172.21.15.157:6837/3388366693", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3388366693, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3388366693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5977, "incarnation": 92, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:rejoin", "state_seq": 14 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:43:10.837009+0000", "qdb_cluster": [ 6246, 5911, 5839, 5517 ], "qdb_leader": 5911, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5977, "mds_2": 5517, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:43:17.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 Processor -- start 2024-04-25T11:43:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 -- start start 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d1006f910 0x7f7d100c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7d100c29f0 0x7f7d1016ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d10160270 con 0x7f7d100c3a90 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7d100c9820 con 0x7f7d100c29f0 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d15861640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100c92a0 con 0x7f7d1006f910 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0e7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7d100c3a90 0x7f7d10073b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:60058/0 (socket says 172.21.15.136:60058) 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0e7fc640 1 -- 172.21.15.136:0/550789502 learned_addr learned my addr 172.21.15.136:0/550789502 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:43:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0f7fe640 1 --2- 172.21.15.136:0/550789502 >> v2:172.21.15.157:3300/0 conn(0x7f7d100c29f0 0x7f7d1016ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0f7fe640 1 -- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d1006f910 msgr2=0x7f7d100c3550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:43:18.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0f7fe640 1 --2- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d1006f910 0x7f7d100c3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0f7fe640 1 -- 172.21.15.136:0/550789502 >> v1:172.21.15.136:6789/0 conn(0x7f7d100c3a90 legacy=0x7f7d10073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.010+0000 7f7d0f7fe640 1 -- 172.21.15.136:0/550789502 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1006ef20 con 0x7f7d100c29f0 2024-04-25T11:43:18.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.018+0000 7f7d0f7fe640 1 --2- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d100c29f0 0x7f7d1016ab10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097c0 tx=0x7f7d0001c2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ab8bb6aa18d91c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.034+0000 7f7d0dffb640 1 -- 172.21.15.136:0/550789502 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d000b0070 con 0x7f7d100c29f0 2024-04-25T11:43:18.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.034+0000 7f7d0dffb640 1 -- 172.21.15.136:0/550789502 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d000ab600 con 0x7f7d100c29f0 2024-04-25T11:43:18.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.034+0000 7f7d0dffb640 1 -- 172.21.15.136:0/550789502 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d000ab9d0 con 0x7f7d100c29f0 2024-04-25T11:43:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d100c29f0 msgr2=0x7f7d1016ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 --2- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d100c29f0 0x7f7d1016ab10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097c0 tx=0x7f7d0001c2c0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- 172.21.15.136:0/550789502 shutdown_connections 2024-04-25T11:43:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 --2- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d100c29f0 0x7f7d1016ab10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 --2- 172.21.15.136:0/550789502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d1006f910 0x7f7d100c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- 172.21.15.136:0/550789502 >> 172.21.15.136:0/550789502 conn(0x7f7d100cdae0 msgr2=0x7f7d100c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:18.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- 172.21.15.136:0/550789502 shutdown_connections 2024-04-25T11:43:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- 172.21.15.136:0/550789502 wait complete. 2024-04-25T11:43:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 Processor -- start 2024-04-25T11:43:18.026 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.038+0000 7f7d15861640 1 -- start start 2024-04-25T11:43:18.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d1006f910 0x7f7d10141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d100c3a90 0x7f7d100cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44842/0 (socket says 172.21.15.136:44842) 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100743b0 con 0x7f7d100c3a90 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10074230 con 0x7f7d1006f910 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7d10074fd0 con 0x7f7d100c29f0 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d100c3a90 0x7f7d100cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 -- 172.21.15.136:0/2814556454 learned_addr learned my addr 172.21.15.136:0/2814556454 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 -- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d1006f910 msgr2=0x7f7d10141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0effd640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d1006f910 0x7f7d10141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d1006f910 0x7f7d10141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 -- 172.21.15.136:0/2814556454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d100c3a90 msgr2=0x7f7d100cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:18.030 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d100c3a90 0x7f7d100cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:18.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 -- 172.21.15.136:0/2814556454 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d10142120 con 0x7f7d100c29f0 2024-04-25T11:43:18.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d0e7fc640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7d100719d0 tx=0x7f7d0800cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08018070 con 0x7f7d100c29f0 2024-04-25T11:43:18.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0800a640 con 0x7f7d100c29f0 2024-04-25T11:43:18.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.042+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d100caf30 con 0x7f7d100c29f0 2024-04-25T11:43:18.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.046+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d100cb1d0 con 0x7f7d100c29f0 2024-04-25T11:43:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.050+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0800aa10 con 0x7f7d100c29f0 2024-04-25T11:43:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.050+0000 7f7cf5ffb640 1 -- 172.21.15.136:0/2814556454 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce80052a0 con 0x7f7d100c29f0 2024-04-25T11:43:18.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.058+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7d0801b070 con 0x7f7d100c29f0 2024-04-25T11:43:18.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.058+0000 7f7cf7fff640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 0x7f7cfc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:18.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.062+0000 7f7d0effd640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 0x7f7cfc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:18.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.062+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f7d0808f330 con 0x7f7d100c29f0 2024-04-25T11:43:18.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.062+0000 7f7d0effd640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 0x7f7cfc071940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a790 tx=0x7f7d04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:18.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:18.062+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7d08091780 con 0x7f7d100c29f0 2024-04-25T11:43:19.101 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.114+0000 7f7cf5ffb640 1 -- 172.21.15.136:0/2814556454 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ce8005590 con 0x7f7d100c29f0 2024-04-25T11:43:19.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.118+0000 7f7cf7fff640 1 -- 172.21.15.136:0/2814556454 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6107 (secure 0 0 0) 0x7f7d0805b350 con 0x7f7d100c29f0 2024-04-25T11:43:19.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:19.119 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 96 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 msgr2=0x7f7cfc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 0x7f7cfc071940 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a790 tx=0x7f7d04008040 comp rx=0 tx=0).stop 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 msgr2=0x7f7d100cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7d100719d0 tx=0x7f7d0800cf50 comp rx=0 tx=0).stop 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 shutdown_connections 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d100c3a90 0x7f7d100cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f7cfc06f430 0x7f7cfc071940 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d100c29f0 0x7f7d100cba00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:19.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 --2- 172.21.15.136:0/2814556454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d1006f910 0x7f7d10141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:19.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 >> 172.21.15.136:0/2814556454 conn(0x7f7d100cdae0 msgr2=0x7f7d1006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:19.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 shutdown_connections 2024-04-25T11:43:19.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:19.134+0000 7f7d15861640 1 -- 172.21.15.136:0/2814556454 wait complete. 2024-04-25T11:43:19.211 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=1, replacing gid=5449 2024-04-25T11:43:19.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.f 2024-04-25T11:43:20.212 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-25T11:43:20.212 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-25T11:43:20.212 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-25T11:43:20.214 INFO:tasks.ceph.mds.f:Started 2024-04-25T11:43:20.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:21.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 Processor -- start 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 -- start start 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f026015a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 0x7f02600707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f02600731c0 0x7f02600735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f0260160760 con 0x7f02600703c0 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0260160a40 con 0x7f02600731c0 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.030+0000 7f0264baf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0260160d20 con 0x7f0260073f60 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f02600731c0 0x7f02600735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:21.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f02600731c0 0x7f02600735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:58514/0 (socket says 172.21.15.136:58514) 2024-04-25T11:43:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 -- 172.21.15.136:0/861582799 learned_addr learned my addr 172.21.15.136:0/861582799 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 -- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 msgr2=0x7f026015a750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:43:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 --2- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f026015a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 -- 172.21.15.136:0/861582799 >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 msgr2=0x7f02600707d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:43:21.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025effd640 1 --2- 172.21.15.136:0/861582799 >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 0x7f02600707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:21.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 --2- 172.21.15.136:0/861582799 >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 0x7f02600707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.034+0000 7f025ffff640 1 -- 172.21.15.136:0/861582799 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0260161000 con 0x7f02600731c0 2024-04-25T11:43:21.023 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.038+0000 7f025effd640 1 --2- 172.21.15.136:0/861582799 >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 0x7f02600707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:21.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.038+0000 7f025ffff640 1 --2- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f02600735d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0260073ef0 tx=0x7f025801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aac501132c81c4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:21.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f025e7fc640 1 -- 172.21.15.136:0/861582799 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02580a56c0 con 0x7f02600731c0 2024-04-25T11:43:21.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f025e7fc640 1 -- 172.21.15.136:0/861582799 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02580a5860 con 0x7f02600731c0 2024-04-25T11:43:21.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f025e7fc640 1 -- 172.21.15.136:0/861582799 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02580a5c70 con 0x7f02600731c0 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 -- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 msgr2=0x7f02600735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 --2- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f02600735d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0260073ef0 tx=0x7f025801ca30 comp rx=0 tx=0).stop 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 -- 172.21.15.136:0/861582799 shutdown_connections 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 --2- 172.21.15.136:0/861582799 >> v2:172.21.15.136:3300/0 conn(0x7f02600703c0 0x7f02600707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 --2- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f02600735d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 --2- 172.21.15.136:0/861582799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f026015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 -- 172.21.15.136:0/861582799 >> 172.21.15.136:0/861582799 conn(0x7f026006cec0 msgr2=0x7f02600cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:21.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 -- 172.21.15.136:0/861582799 shutdown_connections 2024-04-25T11:43:21.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.042+0000 7f0264baf640 1 -- 172.21.15.136:0/861582799 wait complete. 2024-04-25T11:43:21.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.046+0000 7f0264baf640 1 Processor -- start 2024-04-25T11:43:21.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.046+0000 7f0264baf640 1 -- start start 2024-04-25T11:43:21.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.046+0000 7f0264baf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 0x7f0260156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f0264baf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f0260155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f0264baf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f0264baf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02601603b0 con 0x7f02600703c0 2024-04-25T11:43:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44852/0 (socket says 172.21.15.136:44852) 2024-04-25T11:43:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 -- 172.21.15.136:0/1365038658 learned_addr learned my addr 172.21.15.136:0/1365038658 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0260071450 con 0x7f02600731c0 2024-04-25T11:43:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0260156d70 con 0x7f0260073f60 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025f7fe640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 0x7f0260156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 -- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 msgr2=0x7f0260155480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f0260155480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 -- 172.21.15.136:0/1365038658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 msgr2=0x7f0260156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 0x7f0260156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02601568a0 con 0x7f0260073f60 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025f7fe640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 0x7f0260156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:43:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.050+0000 7f025ffff640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f025801b0e0 tx=0x7f02580197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:21.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02580bc020 con 0x7f0260073f60 2024-04-25T11:43:21.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0260158520 con 0x7f0260073f60 2024-04-25T11:43:21.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02601597b0 con 0x7f0260073f60 2024-04-25T11:43:21.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02580b8670 con 0x7f0260073f60 2024-04-25T11:43:21.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02580b8a60 con 0x7f0260073f60 2024-04-25T11:43:21.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02400052a0 con 0x7f0260073f60 2024-04-25T11:43:21.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f02580bf020 con 0x7f0260073f60 2024-04-25T11:43:21.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.054+0000 7f0247fff640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 0x7f024c071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:21.043 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.058+0000 7f025f7fe640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 0x7f024c071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:21.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.058+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f025812a9e0 con 0x7f0260073f60 2024-04-25T11:43:21.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.058+0000 7f025f7fe640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 0x7f024c071960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f025000a830 tx=0x7f0250008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:21.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.066+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f02580c6030 con 0x7f0260073f60 2024-04-25T11:43:21.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.850+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0240005860 con 0x7f0260073f60 2024-04-25T11:43:21.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.858+0000 7f0247fff640 1 -- 172.21.15.136:0/1365038658 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6107 (secure 0 0 0) 0x7f0258016620 con 0x7f0260073f60 2024-04-25T11:43:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:21.843 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5517,5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:43:21.843 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 96 2024-04-25T11:43:21.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 msgr2=0x7f024c071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:21.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 0x7f024c071960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f025000a830 tx=0x7f0250008040 comp rx=0 tx=0).stop 2024-04-25T11:43:21.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 msgr2=0x7f0260148660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:21.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f025801b0e0 tx=0x7f02580197d0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 shutdown_connections 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f024c06f450 0x7f024c071960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0260073f60 0x7f0260148660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02600731c0 0x7f0260155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 --2- 172.21.15.136:0/1365038658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02600703c0 0x7f0260156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 >> 172.21.15.136:0/1365038658 conn(0x7f026006cec0 msgr2=0x7f02600c59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 shutdown_connections 2024-04-25T11:43:21.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:21.874+0000 7f0264baf640 1 -- 172.21.15.136:0/1365038658 wait complete. 2024-04-25T11:43:21.901 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-04-25T11:43:22.010 INFO:tasks.ceph.mds.f.smithi157.stdout:starting mds.f at 2024-04-25T11:43:23.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:43:24.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.398+0000 7fde97eec640 1 Processor -- start 2024-04-25T11:43:24.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.398+0000 7fde97eec640 1 -- start start 2024-04-25T11:43:24.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fde901522f0 0x7fde90070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde900c9370 0x7fde900c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fde900c8ee0 con 0x7fde9015b5d0 2024-04-25T11:43:24.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fde90152700 con 0x7fde901522f0 2024-04-25T11:43:24.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde97eec640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde900729c0 con 0x7fde900c9370 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fde901522f0 0x7fde90070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fde901522f0 0x7fde90070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:58530/0 (socket says 172.21.15.136:58530) 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde96489640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde96489640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50016/0 (socket says 172.21.15.136:50016) 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 -- 172.21.15.136:0/3681395793 learned_addr learned my addr 172.21.15.136:0/3681395793 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:24.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 -- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde900c9370 msgr2=0x7fde900c9780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:43:24.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 --2- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde900c9370 0x7fde900c9780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 -- 172.21.15.136:0/3681395793 >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 msgr2=0x7fde9015b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:43:24.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 --2- 172.21.15.136:0/3681395793 >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.402+0000 7fde95c88640 1 -- 172.21.15.136:0/3681395793 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde90073f30 con 0x7fde901522f0 2024-04-25T11:43:24.392 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.406+0000 7fde96489640 1 --2- 172.21.15.136:0/3681395793 >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:43:24.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.406+0000 7fde95c88640 1 --2- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde901522f0 0x7fde90070700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fde90071ee0 tx=0x7fde8001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=463ac1695df5f1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:24.393 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.406+0000 7fde94c86640 1 -- 172.21.15.136:0/3681395793 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde800b1070 con 0x7fde901522f0 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.406+0000 7fde94c86640 1 -- 172.21.15.136:0/3681395793 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde800ac600 con 0x7fde901522f0 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 -- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde901522f0 msgr2=0x7fde90070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde94c86640 1 -- 172.21.15.136:0/3681395793 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde800b1070 con 0x7fde901522f0 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 --2- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde901522f0 0x7fde90070700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fde90071ee0 tx=0x7fde8001c4d0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 -- 172.21.15.136:0/3681395793 shutdown_connections 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 --2- 172.21.15.136:0/3681395793 >> v2:172.21.15.136:3300/0 conn(0x7fde9015b5d0 0x7fde9015b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 --2- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde900c9370 0x7fde900c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 --2- 172.21.15.136:0/3681395793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde901522f0 0x7fde90070700 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.410+0000 7fde97eec640 1 -- 172.21.15.136:0/3681395793 >> 172.21.15.136:0/3681395793 conn(0x7fde900c9d50 msgr2=0x7fde900ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:24.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.414+0000 7fde97eec640 1 -- 172.21.15.136:0/3681395793 shutdown_connections 2024-04-25T11:43:24.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.414+0000 7fde97eec640 1 -- 172.21.15.136:0/3681395793 wait complete. 2024-04-25T11:43:24.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 Processor -- start 2024-04-25T11:43:24.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 -- start start 2024-04-25T11:43:24.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde900c9370 0x7fde901453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde9015b5d0 0x7fde9013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde900ca9f0 con 0x7fde90143c10 2024-04-25T11:43:24.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde900ca870 con 0x7fde900c9370 2024-04-25T11:43:24.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.418+0000 7fde97eec640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde901587f0 con 0x7fde9015b5d0 2024-04-25T11:43:24.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:24.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50032/0 (socket says 172.21.15.136:50032) 2024-04-25T11:43:24.412 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 -- 172.21.15.136:0/65259640 learned_addr learned my addr 172.21.15.136:0/65259640 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 -- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde9015b5d0 msgr2=0x7fde9013ed90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde9015b5d0 0x7fde9013ed90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 -- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde900c9370 msgr2=0x7fde901453a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde900c9370 0x7fde901453a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 -- 172.21.15.136:0/65259640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde90141a00 con 0x7fde90143c10 2024-04-25T11:43:24.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.426+0000 7fde96489640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00eb60 tx=0x7fde8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:24.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.430+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c017070 con 0x7fde90143c10 2024-04-25T11:43:24.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.430+0000 7fde97eec640 1 -- 172.21.15.136:0/65259640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde90144b50 con 0x7fde90143c10 2024-04-25T11:43:24.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.430+0000 7fde97eec640 1 -- 172.21.15.136:0/65259640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9013e590 con 0x7fde90143c10 2024-04-25T11:43:24.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.434+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8c00c8e0 con 0x7fde90143c10 2024-04-25T11:43:24.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.434+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c00ccb0 con 0x7fde90143c10 2024-04-25T11:43:24.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.438+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde8c01a070 con 0x7fde90143c10 2024-04-25T11:43:24.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.438+0000 7fde867fc640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 0x7fde740719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:43:24.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.438+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fde8c08fca0 con 0x7fde90143c10 2024-04-25T11:43:24.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.438+0000 7fde95c88640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 0x7fde740719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:43:24.425 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.438+0000 7fde95c88640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 0x7fde740719e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fde800ae000 tx=0x7fde80004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:43:24.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.462+0000 7fde97eec640 1 -- 172.21.15.136:0/65259640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde780052a0 con 0x7fde90143c10 2024-04-25T11:43:24.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:24.490+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde8c05c3a0 con 0x7fde90143c10 2024-04-25T11:43:25.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.210+0000 7fde97eec640 1 -- 172.21.15.136:0/65259640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde78005590 con 0x7fde90143c10 2024-04-25T11:43:25.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.214+0000 7fde867fc640 1 -- 172.21.15.136:0/65259640 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7fde8c05bca0 con 0x7fde90143c10 2024-04-25T11:43:25.204 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:43:25.204 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:43:25.204 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:43:25.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.222+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 msgr2=0x7fde740719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:25.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.222+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 0x7fde740719e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fde800ae000 tx=0x7fde80004740 comp rx=0 tx=0).stop 2024-04-25T11:43:25.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.222+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 msgr2=0x7fde9013fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:43:25.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.222+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00eb60 tx=0x7fde8c00ef60 comp rx=0 tx=0).stop 2024-04-25T11:43:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 shutdown_connections 2024-04-25T11:43:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde90143c10 0x7fde9013fe80 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fde7406f4d0 0x7fde740719e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde9015b5d0 0x7fde9013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:25.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 --2- 172.21.15.136:0/65259640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde900c9370 0x7fde901453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:43:25.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.226+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 >> 172.21.15.136:0/65259640 conn(0x7fde900c9d50 msgr2=0x7fde9006e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:43:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.234+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 shutdown_connections 2024-04-25T11:43:25.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:43:25.238+0000 7fde73fff640 1 -- 172.21.15.136:0/65259640 wait complete. 2024-04-25T11:43:25.270 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-25T11:43:25.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2024-04-25T11:43:29.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:43:29.775 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T11:43:30.049 INFO:tasks.ceph.osd.1.smithi136.stderr:2024-04-25T11:43:30.061+0000 7fcd2539d640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 57889) UID: 0 2024-04-25T11:43:30.050 INFO:tasks.ceph.osd.3.smithi136.stderr:2024-04-25T11:43:30.061+0000 7fb512277640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 57889) UID: 0 2024-04-25T11:43:30.052 INFO:tasks.ceph.osd.2.smithi136.stderr:2024-04-25T11:43:30.061+0000 7fd0f4edc640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 57889) UID: 0 2024-04-25T11:43:30.060 INFO:tasks.ceph.osd.0.smithi136.stderr:2024-04-25T11:43:30.069+0000 7eff4f7bf640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 57889) UID: 0 2024-04-25T11:44:03.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.649+0000 7fda81706640 1 Processor -- start 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.649+0000 7fda81706640 1 -- start start 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda81706640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 0x7fda7c16cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda81706640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda7c0c7b30 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda81706640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda7c0c9630 con 0x7fda7c073300 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda81706640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fda7c0c93e0 con 0x7fda7c16c870 2024-04-25T11:44:03.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7affd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fda7c073300 0x7fda7c0c85d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:48728/0 (socket says 172.21.15.136:48728) 2024-04-25T11:44:03.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7affd640 1 -- 172.21.15.136:0/1404952516 learned_addr learned my addr 172.21.15.136:0/1404952516 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:03.641 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7b7fe640 1 --2- 172.21.15.136:0/1404952516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 0x7fda7c16cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda79ffb640 1 -- 172.21.15.136:0/1404952516 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665200170 0 0) 0x7fda7c0c9630 con 0x7fda7c073300 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda79ffb640 1 -- 172.21.15.136:0/1404952516 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda64003510 con 0x7fda7c073300 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7b7fe640 1 -- 172.21.15.136:0/1404952516 >> v1:172.21.15.157:6789/0 conn(0x7fda7c073300 legacy=0x7fda7c0c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7b7fe640 1 -- 172.21.15.136:0/1404952516 >> v1:172.21.15.136:6789/0 conn(0x7fda7c0c2bb0 legacy=0x7fda7c0c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.653+0000 7fda7b7fe640 1 -- 172.21.15.136:0/1404952516 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c0c7dd0 con 0x7fda7c16c870 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.657+0000 7fda7b7fe640 1 --2- 172.21.15.136:0/1404952516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 0x7fda7c16cc80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda7c06f7c0 tx=0x7fda700249c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c2fd7e7a17cdcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.657+0000 7fda79ffb640 1 -- 172.21.15.136:0/1404952516 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c001680 con 0x7fda7c16c870 2024-04-25T11:44:03.642 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.657+0000 7fda79ffb640 1 -- 172.21.15.136:0/1404952516 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda70009130 con 0x7fda7c16c870 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.661+0000 7fda81706640 1 -- 172.21.15.136:0/1404952516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 msgr2=0x7fda7c16cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.661+0000 7fda81706640 1 --2- 172.21.15.136:0/1404952516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 0x7fda7c16cc80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda7c06f7c0 tx=0x7fda700249c0 comp rx=0 tx=0).stop 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 -- 172.21.15.136:0/1404952516 shutdown_connections 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 --2- 172.21.15.136:0/1404952516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c16c870 0x7fda7c16cc80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 -- 172.21.15.136:0/1404952516 >> 172.21.15.136:0/1404952516 conn(0x7fda7c0cdae0 msgr2=0x7fda7c0c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:03.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 -- 172.21.15.136:0/1404952516 shutdown_connections 2024-04-25T11:44:03.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 -- 172.21.15.136:0/1404952516 wait complete. 2024-04-25T11:44:03.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.665+0000 7fda81706640 1 Processor -- start 2024-04-25T11:44:03.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 -- start start 2024-04-25T11:44:03.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c073300 0x7fda7c0c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:03.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:03.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda7c16c870 0x7fda7c16bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:03.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c0c8eb0 con 0x7fda7c16c870 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda7c0c2990 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.673+0000 7fda81706640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fda7c073a50 con 0x7fda7c073300 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:54930/0 (socket says 172.21.15.136:54930) 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 -- 172.21.15.136:0/2693130491 learned_addr learned my addr 172.21.15.136:0/2693130491 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7affd640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c073300 0x7fda7c0c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 -- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c073300 msgr2=0x7fda7c0c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c073300 0x7fda7c0c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 -- 172.21.15.136:0/2693130491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda7c16c870 msgr2=0x7fda7c16bc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda7c16c870 0x7fda7c16bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 -- 172.21.15.136:0/2693130491 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c0c6820 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda7a7fc640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda7400ba10 tx=0x7fda7400bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:03.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74019070 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda81706640 1 -- 172.21.15.136:0/2693130491 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7c206a20 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.677+0000 7fda81706640 1 -- 172.21.15.136:0/2693130491 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda7c206750 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.681+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7400cbd0 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.681+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74014440 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.681+0000 7fda81706640 1 -- 172.21.15.136:0/2693130491 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda7c06a2e0 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.697+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda7401c070 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.697+0000 7fda637fe640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 0x7fda68071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:03.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.701+0000 7fda7affd640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 0x7fda68071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:03.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.705+0000 7fda7affd640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 0x7fda68071a10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0017f0 tx=0x7fda6c001a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:03.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.705+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fda7408fa00 con 0x7fda7c0c2bb0 2024-04-25T11:44:03.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:03.709+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fda740bbe70 con 0x7fda7c0c2bb0 2024-04-25T11:44:04.487 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.505+0000 7fda81706640 1 -- 172.21.15.136:0/2693130491 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda7c06d310 con 0x7fda7c0c2bb0 2024-04-25T11:44:04.488 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.505+0000 7fda637fe640 1 -- 172.21.15.136:0/2693130491 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7fda7405bab0 con 0x7fda7c0c2bb0 2024-04-25T11:44:04.489 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:04.489 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:04.490 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:04.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 msgr2=0x7fda68071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:04.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 0x7fda68071a10 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0017f0 tx=0x7fda6c001a90 comp rx=0 tx=0).stop 2024-04-25T11:44:04.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 msgr2=0x7fda7c0c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:04.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fda7400ba10 tx=0x7fda7400bed0 comp rx=0 tx=0).stop 2024-04-25T11:44:04.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 shutdown_connections 2024-04-25T11:44:04.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda7c16c870 0x7fda7c16bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:04.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda7c0c2bb0 0x7fda7c0c62e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:04.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fda6806f500 0x7fda68071a10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:04.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 --2- 172.21.15.136:0/2693130491 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda7c073300 0x7fda7c0c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:04.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 >> 172.21.15.136:0/2693130491 conn(0x7fda7c0cdae0 msgr2=0x7fda7c168710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:04.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.521+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 shutdown_connections 2024-04-25T11:44:04.509 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:04.525+0000 7fda617fa640 1 -- 172.21.15.136:0/2693130491 wait complete. 2024-04-25T11:44:04.590 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2024-04-25T11:44:04.590 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-25T11:44:04.591 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:44:04.647 INFO:tasks.ceph.mds.e:Stopped 2024-04-25T11:44:04.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:05.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.105+0000 7f795d2a2640 1 Processor -- start 2024-04-25T11:44:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.105+0000 7f795d2a2640 1 -- start start 2024-04-25T11:44:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f795d2a2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f795004c8d0 0x7f79500ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f795d2a2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79500e4480 con 0x7f7950045390 2024-04-25T11:44:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f795d2a2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7950047560 con 0x7f79500ddc70 2024-04-25T11:44:05.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f795d2a2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f79500461f0 con 0x7f795004c8d0 2024-04-25T11:44:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79567fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f79500ddc70 0x7f79500062a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:43844/0 (socket says 172.21.15.136:43844) 2024-04-25T11:44:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79567fc640 1 -- 172.21.15.136:0/4184445608 learned_addr learned my addr 172.21.15.136:0/4184445608 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:05.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79577fe640 1 --2- 172.21.15.136:0/4184445608 >> v2:172.21.15.157:3300/0 conn(0x7f795004c8d0 0x7f79500ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79577fe640 1 -- 172.21.15.136:0/4184445608 >> v1:172.21.15.157:6789/0 conn(0x7f79500ddc70 legacy=0x7f79500062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79577fe640 1 -- 172.21.15.136:0/4184445608 >> v1:172.21.15.136:6789/0 conn(0x7f7950045390 legacy=0x7f7950046e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:05.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.109+0000 7f79577fe640 1 -- 172.21.15.136:0/4184445608 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7950046390 con 0x7f795004c8d0 2024-04-25T11:44:05.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.113+0000 7f79577fe640 1 --2- 172.21.15.136:0/4184445608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500ef1f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7948009740 tx=0x7f794801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c887a473622523 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f7955ffb640 1 -- 172.21.15.136:0/4184445608 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79480b2070 con 0x7f795004c8d0 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f7955ffb640 1 -- 172.21.15.136:0/4184445608 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79480a9070 con 0x7f795004c8d0 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f7955ffb640 1 -- 172.21.15.136:0/4184445608 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79480ad880 con 0x7f795004c8d0 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f795d2a2640 1 -- 172.21.15.136:0/4184445608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 msgr2=0x7f79500ef1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f795d2a2640 1 --2- 172.21.15.136:0/4184445608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500ef1f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7948009740 tx=0x7f794801c300 comp rx=0 tx=0).stop 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f795d2a2640 1 -- 172.21.15.136:0/4184445608 shutdown_connections 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f795d2a2640 1 --2- 172.21.15.136:0/4184445608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500ef1f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.121+0000 7f795d2a2640 1 -- 172.21.15.136:0/4184445608 >> 172.21.15.136:0/4184445608 conn(0x7f79500503f0 msgr2=0x7f7950050800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:05.128 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.145+0000 7f795d2a2640 1 -- 172.21.15.136:0/4184445608 shutdown_connections 2024-04-25T11:44:05.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.145+0000 7f795d2a2640 1 -- 172.21.15.136:0/4184445608 wait complete. 2024-04-25T11:44:05.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 Processor -- start 2024-04-25T11:44:05.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 -- start start 2024-04-25T11:44:05.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:05.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500da740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:05.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79500ddc70 0x7f79500e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:05.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f79500e4720 con 0x7f7950045390 2024-04-25T11:44:05.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f79500469c0 con 0x7f795004c8d0 2024-04-25T11:44:05.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f795d2a2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7950046650 con 0x7f79500ddc70 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f7956ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f7956ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44400/0 (socket says 172.21.15.136:44400) 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f7956ffd640 1 -- 172.21.15.136:0/1756664003 learned_addr learned my addr 172.21.15.136:0/1756664003 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.149+0000 7f79567fc640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500da740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 -- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79500ddc70 msgr2=0x7f79500e2300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79500ddc70 0x7f79500e2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 -- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 msgr2=0x7f79500da740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500da740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 -- 172.21.15.136:0/1756664003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7950035860 con 0x7f7950045390 2024-04-25T11:44:05.136 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.153+0000 7f7956ffd640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f795806b4c0 tx=0x7f795806aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:05.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958091070 con 0x7f7950045390 2024-04-25T11:44:05.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79500e3480 con 0x7f7950045390 2024-04-25T11:44:05.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79500e3a40 con 0x7f7950045390 2024-04-25T11:44:05.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795806dbd0 con 0x7f7950045390 2024-04-25T11:44:05.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580955b0 con 0x7f7950045390 2024-04-25T11:44:05.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.165+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7958094070 con 0x7f7950045390 2024-04-25T11:44:05.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.169+0000 7f793f7fe640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 0x7f7940071b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:05.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.169+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f795810c250 con 0x7f7950045390 2024-04-25T11:44:05.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.169+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79380052a0 con 0x7f7950045390 2024-04-25T11:44:05.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.173+0000 7f79567fc640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 0x7f7940071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:05.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.177+0000 7f79567fc640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 0x7f7940071b20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f794c009fd0 tx=0x7f794c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:05.164 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.181+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f79580d8270 con 0x7f7950045390 2024-04-25T11:44:05.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.917+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7938005590 con 0x7f7950045390 2024-04-25T11:44:05.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.917+0000 7f793f7fe640 1 -- 172.21.15.136:0/1756664003 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7f79580dc570 con 0x7f7950045390 2024-04-25T11:44:05.906 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:05.906 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.921+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 msgr2=0x7f7940071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.921+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 0x7f7940071b20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f794c009fd0 tx=0x7f794c00b040 comp rx=0 tx=0).stop 2024-04-25T11:44:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.925+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 msgr2=0x7f79500e2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.925+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f795806b4c0 tx=0x7f795806aca0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 shutdown_connections 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f794006f610 0x7f7940071b20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79500ddc70 0x7f79500e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f795004c8d0 0x7f79500da740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 --2- 172.21.15.136:0/1756664003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7950045390 0x7f79500e2a80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 >> 172.21.15.136:0/1756664003 conn(0x7f79500503f0 msgr2=0x7f79500d81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 shutdown_connections 2024-04-25T11:44:05.917 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:05.933+0000 7f795d2a2640 1 -- 172.21.15.136:0/1756664003 wait complete. 2024-04-25T11:44:05.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:44:07.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:08.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 Processor -- start 2024-04-25T11:44:08.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 -- start start 2024-04-25T11:44:08.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6680c2470 0x7fd6680c9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:08.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd66815be80 con 0x7fd6680c9c70 2024-04-25T11:44:08.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd66815c3e0 con 0x7fd6681521f0 2024-04-25T11:44:08.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd667fff640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd668152600 con 0x7fd6680c2470 2024-04-25T11:44:08.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd6677fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd6680c9c70 0x7fd668162830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:41060/0 (socket says 172.21.15.136:41060) 2024-04-25T11:44:08.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.373+0000 7fd6677fe640 1 -- 172.21.15.136:0/2574075268 learned_addr learned my addr 172.21.15.136:0/2574075268 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:08.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.377+0000 7fd65e7fc640 1 --2- 172.21.15.136:0/2574075268 >> v2:172.21.15.136:3300/0 conn(0x7fd6680c2470 0x7fd6680c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:08.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.377+0000 7fd65e7fc640 1 -- 172.21.15.136:0/2574075268 >> v1:172.21.15.157:6789/0 conn(0x7fd6681521f0 legacy=0x7fd668158570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:08.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.377+0000 7fd65e7fc640 1 -- 172.21.15.136:0/2574075268 >> v1:172.21.15.136:6789/0 conn(0x7fd6680c9c70 legacy=0x7fd668162830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.377+0000 7fd65e7fc640 1 -- 172.21.15.136:0/2574075268 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66815c100 con 0x7fd6680c2470 2024-04-25T11:44:08.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.381+0000 7fd65e7fc640 1 --2- 172.21.15.136:0/2574075268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c2470 0x7fd6680c9730 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd6500097c0 tx=0x7fd65001e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdc9fdd1b94f0dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:08.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.409+0000 7fd6667fc640 1 -- 172.21.15.136:0/2574075268 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6500ac5f0 con 0x7fd6680c2470 2024-04-25T11:44:08.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.409+0000 7fd6667fc640 1 -- 172.21.15.136:0/2574075268 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6500b1070 con 0x7fd6680c2470 2024-04-25T11:44:08.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 -- 172.21.15.136:0/2574075268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c2470 msgr2=0x7fd6680c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 --2- 172.21.15.136:0/2574075268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c2470 0x7fd6680c9730 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd6500097c0 tx=0x7fd65001e680 comp rx=0 tx=0).stop 2024-04-25T11:44:08.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 -- 172.21.15.136:0/2574075268 shutdown_connections 2024-04-25T11:44:08.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 --2- 172.21.15.136:0/2574075268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c2470 0x7fd6680c9730 secure :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd6500097c0 tx=0x7fd65001e680 comp rx=0 tx=0).stop 2024-04-25T11:44:08.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 -- 172.21.15.136:0/2574075268 >> 172.21.15.136:0/2574075268 conn(0x7fd66806d100 msgr2=0x7fd668070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:08.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 -- 172.21.15.136:0/2574075268 shutdown_connections 2024-04-25T11:44:08.397 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.413+0000 7fd667fff640 1 -- 172.21.15.136:0/2574075268 wait complete. 2024-04-25T11:44:08.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 Processor -- start 2024-04-25T11:44:08.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 -- start start 2024-04-25T11:44:08.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:08.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6681521f0 0x7fd66814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:08.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 0x7fd668141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:08.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd668158c80 con 0x7fd6680c9c70 2024-04-25T11:44:08.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd6680c2880 con 0x7fd668141250 2024-04-25T11:44:08.403 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.417+0000 7fd667fff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd668158e00 con 0x7fd6681521f0 2024-04-25T11:44:08.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:08.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd6677fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 0x7fd668141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:08.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44424/0 (socket says 172.21.15.136:44424) 2024-04-25T11:44:08.404 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 -- 172.21.15.136:0/4170776714 learned_addr learned my addr 172.21.15.136:0/4170776714 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd6677fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 0x7fd668141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52374/0 (socket says 172.21.15.136:52374) 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 -- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6681521f0 msgr2=0x7fd66814f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6681521f0 0x7fd66814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 -- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 msgr2=0x7fd668141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 0x7fd668141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 -- 172.21.15.136:0/4170776714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6681515c0 con 0x7fd6680c9c70 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd666ffd640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd65800f7e0 tx=0x7fd65800fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800d070 con 0x7fd6680c9c70 2024-04-25T11:44:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6681f2070 con 0x7fd6680c9c70 2024-04-25T11:44:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.421+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6681f2630 con 0x7fd6680c9c70 2024-04-25T11:44:08.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd658007840 con 0x7fd6680c9c70 2024-04-25T11:44:08.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658007c30 con 0x7fd6680c9c70 2024-04-25T11:44:08.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6580163b0 con 0x7fd6680c9c70 2024-04-25T11:44:08.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd664ff9640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 0x7fd648071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:08.408 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7fd65808f0e0 con 0x7fd6680c9c70 2024-04-25T11:44:08.409 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.425+0000 7fd65e7fc640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 0x7fd648071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:08.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.429+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64c0052a0 con 0x7fd6680c9c70 2024-04-25T11:44:08.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.433+0000 7fd65e7fc640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 0x7fd648071a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd6500b0000 tx=0x7fd650004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:08.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.441+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd65805b100 con 0x7fd6680c9c70 2024-04-25T11:44:08.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.885+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd64c005590 con 0x7fd6680c9c70 2024-04-25T11:44:08.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.885+0000 7fd664ff9640 1 -- 172.21.15.136:0/4170776714 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7fd65805f400 con 0x7fd6680c9c70 2024-04-25T11:44:08.870 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:08.870 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:08.870 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:08.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 msgr2=0x7fd648071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 0x7fd648071a40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd6500b0000 tx=0x7fd650004740 comp rx=0 tx=0).stop 2024-04-25T11:44:08.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 msgr2=0x7fd668146370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:08.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd65800f7e0 tx=0x7fd65800fca0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 shutdown_connections 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd668141250 0x7fd668141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd64806f530 0x7fd648071a40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd6681521f0 0x7fd66814f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 --2- 172.21.15.136:0/4170776714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6680c9c70 0x7fd668146370 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 >> 172.21.15.136:0/4170776714 conn(0x7fd66806d100 msgr2=0x7fd66815a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 shutdown_connections 2024-04-25T11:44:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:08.905+0000 7fd667fff640 1 -- 172.21.15.136:0/4170776714 wait complete. 2024-04-25T11:44:08.941 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:44:10.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:11.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.325+0000 7f6a8da95640 1 Processor -- start 2024-04-25T11:44:11.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.325+0000 7f6a8da95640 1 -- start start 2024-04-25T11:44:11.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a8da95640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a8da95640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a880732e0 0x7f6a881757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a8da95640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a880c92a0 con 0x7f6a880727c0 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a8da95640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6a880c9820 con 0x7f6a880732e0 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a8da95640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8806fd30 con 0x7f6a88074220 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:41022/0 (socket says 172.21.15.136:41022) 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 -- 172.21.15.136:0/548032859 learned_addr learned my addr 172.21.15.136:0/548032859 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a87fff640 1 --2- 172.21.15.136:0/548032859 >> v2:172.21.15.157:3300/0 conn(0x7f6a880732e0 0x7f6a881757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 -- 172.21.15.136:0/548032859 >> v2:172.21.15.157:3300/0 conn(0x7f6a880732e0 msgr2=0x7f6a881757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 --2- 172.21.15.136:0/548032859 >> v2:172.21.15.157:3300/0 conn(0x7f6a880732e0 0x7f6a881757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 -- 172.21.15.136:0/548032859 >> v1:172.21.15.157:6789/0 conn(0x7f6a880727c0 legacy=0x7f6a88072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.329+0000 7f6a877fe640 1 -- 172.21.15.136:0/548032859 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a880c9540 con 0x7f6a88074220 2024-04-25T11:44:11.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.333+0000 7f6a877fe640 1 --2- 172.21.15.136:0/548032859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a7801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92c46784edf2587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:11.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.341+0000 7f6a86ffd640 1 -- 172.21.15.136:0/548032859 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a780b1070 con 0x7f6a88074220 2024-04-25T11:44:11.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.341+0000 7f6a86ffd640 1 -- 172.21.15.136:0/548032859 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a780a8050 con 0x7f6a88074220 2024-04-25T11:44:11.325 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.341+0000 7f6a86ffd640 1 -- 172.21.15.136:0/548032859 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a780ac860 con 0x7f6a88074220 2024-04-25T11:44:11.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.345+0000 7f6a8da95640 1 -- 172.21.15.136:0/548032859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 msgr2=0x7f6a8816ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.345+0000 7f6a8da95640 1 --2- 172.21.15.136:0/548032859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a7801c480 comp rx=0 tx=0).stop 2024-04-25T11:44:11.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.353+0000 7f6a8da95640 1 -- 172.21.15.136:0/548032859 shutdown_connections 2024-04-25T11:44:11.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.353+0000 7f6a8da95640 1 --2- 172.21.15.136:0/548032859 >> v2:172.21.15.157:3300/0 conn(0x7f6a880732e0 0x7f6a881757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.353+0000 7f6a8da95640 1 --2- 172.21.15.136:0/548032859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a88074220 0x7f6a8816ed40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.353+0000 7f6a8da95640 1 -- 172.21.15.136:0/548032859 >> 172.21.15.136:0/548032859 conn(0x7f6a880cdae0 msgr2=0x7f6a880c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:11.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.373+0000 7f6a8da95640 1 -- 172.21.15.136:0/548032859 shutdown_connections 2024-04-25T11:44:11.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.373+0000 7f6a8da95640 1 -- 172.21.15.136:0/548032859 wait complete. 2024-04-25T11:44:11.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.377+0000 7f6a8da95640 1 Processor -- start 2024-04-25T11:44:11.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.377+0000 7f6a8da95640 1 -- start start 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.377+0000 7f6a8da95640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a880727c0 0x7f6a88163c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 0x7f6a88167570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8806eef0 con 0x7f6a880732e0 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88073d00 con 0x7f6a88074220 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8816f280 con 0x7f6a880727c0 2024-04-25T11:44:11.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a7ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a877fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a880727c0 0x7f6a88163c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a7ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44430/0 (socket says 172.21.15.136:44430) 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a7ffff640 1 -- 172.21.15.136:0/2572067625 learned_addr learned my addr 172.21.15.136:0/2572067625 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 0x7f6a88167570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 -- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a880727c0 msgr2=0x7f6a88163c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a880727c0 0x7f6a88163c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 -- 172.21.15.136:0/2572067625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 msgr2=0x7f6a88168e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 -- 172.21.15.136:0/2572067625 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a88165470 con 0x7f6a88074220 2024-04-25T11:44:11.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a7ffff640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:11.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a87fff640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 0x7f6a88167570 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6a80007cf0 tx=0x7f6a8000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:11.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a800039a0 con 0x7f6a88074220 2024-04-25T11:44:11.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881693a0 con 0x7f6a88074220 2024-04-25T11:44:11.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a880cc3e0 con 0x7f6a88074220 2024-04-25T11:44:11.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a80003b60 con 0x7f6a88074220 2024-04-25T11:44:11.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.381+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80004020 con 0x7f6a88074220 2024-04-25T11:44:11.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.385+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6a800041c0 con 0x7f6a88074220 2024-04-25T11:44:11.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.385+0000 7f6a84ff9640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 0x7f6a68071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:11.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.385+0000 7f6a877fe640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 0x7f6a68071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:11.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.385+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f6a80093e90 con 0x7f6a88074220 2024-04-25T11:44:11.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.385+0000 7f6a877fe640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 0x7f6a68071940 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6a780ae000 tx=0x7f6a780a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:11.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.393+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6c0052a0 con 0x7f6a88074220 2024-04-25T11:44:11.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:11.409+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6a80060590 con 0x7f6a88074220 2024-04-25T11:44:12.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.069+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a6c005860 con 0x7f6a88074220 2024-04-25T11:44:12.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.081+0000 7f6a84ff9640 1 -- 172.21.15.136:0/2572067625 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7f6a8005fe90 con 0x7f6a88074220 2024-04-25T11:44:12.068 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:12.069 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:12.069 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 msgr2=0x7f6a68071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 0x7f6a68071940 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6a780ae000 tx=0x7f6a780a7000 comp rx=0 tx=0).stop 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 msgr2=0x7f6a88167570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 0x7f6a88167570 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6a80007cf0 tx=0x7f6a8000e4b0 comp rx=0 tx=0).stop 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 shutdown_connections 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a88074220 0x7f6a88167570 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a880732e0 0x7f6a88168e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:12.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f6a6806f430 0x7f6a68071940 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:12.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 --2- 172.21.15.136:0/2572067625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a880727c0 0x7f6a88163c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:12.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.089+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 >> 172.21.15.136:0/2572067625 conn(0x7f6a880cdae0 msgr2=0x7f6a88075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:12.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.101+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 shutdown_connections 2024-04-25T11:44:12.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:12.101+0000 7f6a8da95640 1 -- 172.21.15.136:0/2572067625 wait complete. 2024-04-25T11:44:12.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:44:14.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:14.705 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.720+0000 7f4b4bb07640 1 Processor -- start 2024-04-25T11:44:14.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.720+0000 7f4b4bb07640 1 -- start start 2024-04-25T11:44:14.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b4bb07640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4b440c3010 0x7f4b4406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b4bb07640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b4bb07640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b4406ea50 con 0x7f4b440afb80 2024-04-25T11:44:14.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b4bb07640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f4b4406ecc0 con 0x7f4b44152360 2024-04-25T11:44:14.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b4bb07640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4b440c3640 con 0x7f4b440c3010 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b490a2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b490a2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46242/0 (socket says 172.21.15.136:46242) 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b490a2640 1 -- 172.21.15.136:0/1440082262 learned_addr learned my addr 172.21.15.136:0/1440082262 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b498a3640 1 --2- 172.21.15.136:0/1440082262 >> v2:172.21.15.157:3300/0 conn(0x7f4b440c3010 0x7f4b4406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b498a3640 1 -- 172.21.15.136:0/1440082262 >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 msgr2=0x7f4b440af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b498a3640 1 --2- 172.21.15.136:0/1440082262 >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.724+0000 7f4b498a3640 1 -- 172.21.15.136:0/1440082262 >> v1:172.21.15.136:6789/0 conn(0x7f4b440afb80 legacy=0x7f4b44157ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:14.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.744+0000 7f4b498a3640 1 -- 172.21.15.136:0/1440082262 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b440c3920 con 0x7f4b440c3010 2024-04-25T11:44:14.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.744+0000 7f4b490a2640 1 --2- 172.21.15.136:0/1440082262 >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:14.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.744+0000 7f4b498a3640 1 --2- 172.21.15.136:0/1440082262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b440c3010 0x7f4b4406e510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4b44072270 tx=0x7f4b40024bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ed3e4855730c05d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:14.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b488a1640 1 -- 172.21.15.136:0/1440082262 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000d2d0 con 0x7f4b440c3010 2024-04-25T11:44:14.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b488a1640 1 -- 172.21.15.136:0/1440082262 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b400ad890 con 0x7f4b440c3010 2024-04-25T11:44:14.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b488a1640 1 -- 172.21.15.136:0/1440082262 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b400adca0 con 0x7f4b440c3010 2024-04-25T11:44:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 -- 172.21.15.136:0/1440082262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b440c3010 msgr2=0x7f4b4406e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/1440082262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b440c3010 0x7f4b4406e510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4b44072270 tx=0x7f4b40024bc0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 -- 172.21.15.136:0/1440082262 shutdown_connections 2024-04-25T11:44:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/1440082262 >> v2:172.21.15.136:3300/0 conn(0x7f4b44152360 0x7f4b440af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/1440082262 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b440c3010 0x7f4b4406e510 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.748+0000 7f4b4bb07640 1 -- 172.21.15.136:0/1440082262 >> 172.21.15.136:0/1440082262 conn(0x7f4b440cdae0 msgr2=0x7f4b440c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:14.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.768+0000 7f4b4bb07640 1 -- 172.21.15.136:0/1440082262 shutdown_connections 2024-04-25T11:44:14.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.768+0000 7f4b4bb07640 1 -- 172.21.15.136:0/1440082262 wait complete. 2024-04-25T11:44:14.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 Processor -- start 2024-04-25T11:44:14.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 -- start start 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b440afb80 0x7f4b441576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b44152360 0x7f4b441ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44165310 con 0x7f4b440c3010 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44165190 con 0x7f4b44152360 2024-04-25T11:44:14.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b4bb07640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4b44165490 con 0x7f4b440afb80 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b498a3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b440afb80 0x7f4b441576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b490a2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b490a2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46256/0 (socket says 172.21.15.136:46256) 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.772+0000 7f4b490a2640 1 -- 172.21.15.136:0/3743049111 learned_addr learned my addr 172.21.15.136:0/3743049111 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 -- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b440afb80 msgr2=0x7f4b441576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b440afb80 0x7f4b441576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 -- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b44152360 msgr2=0x7f4b441ffd90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b44152360 0x7f4b441ffd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 -- 172.21.15.136:0/3743049111 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b440b0300 con 0x7f4b440c3010 2024-04-25T11:44:14.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.776+0000 7f4b490a2640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c007fd0 tx=0x7f4b3c00e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:14.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.796+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00be60 con 0x7f4b440c3010 2024-04-25T11:44:14.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.796+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b440b1890 con 0x7f4b440c3010 2024-04-25T11:44:14.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.796+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b440b1340 con 0x7f4b440c3010 2024-04-25T11:44:14.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.808+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3c003ba0 con 0x7f4b440c3010 2024-04-25T11:44:14.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.812+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c004030 con 0x7f4b440c3010 2024-04-25T11:44:14.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.812+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4b3c004210 con 0x7f4b440c3010 2024-04-25T11:44:14.795 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.812+0000 7f4b3a7fc640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 0x7f4b28071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:14.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.812+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f4b3c090570 con 0x7f4b440c3010 2024-04-25T11:44:14.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.812+0000 7f4b498a3640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 0x7f4b28071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:14.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.820+0000 7f4b498a3640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 0x7f4b28071a40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4b400b6000 tx=0x7f4b4000f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:14.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.820+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b2c0052a0 con 0x7f4b440c3010 2024-04-25T11:44:14.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:14.832+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4b3c05cc70 con 0x7f4b440c3010 2024-04-25T11:44:15.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.644+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b2c005590 con 0x7f4b440c3010 2024-04-25T11:44:15.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.644+0000 7f4b3a7fc640 1 -- 172.21.15.136:0/3743049111 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7f4b3c05c570 con 0x7f4b440c3010 2024-04-25T11:44:15.630 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:15.630 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:15.630 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:15.645 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.660+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 msgr2=0x7f4b28071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.660+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 0x7f4b28071a40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4b400b6000 tx=0x7f4b4000f040 comp rx=0 tx=0).stop 2024-04-25T11:44:15.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 msgr2=0x7f4b441ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:15.648 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c007fd0 tx=0x7f4b3c00e5b0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 shutdown_connections 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b44152360 0x7f4b441ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b440c3010 0x7f4b441ff850 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f4b2806f530 0x7f4b28071a40 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 --2- 172.21.15.136:0/3743049111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4b440afb80 0x7f4b441576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 >> 172.21.15.136:0/3743049111 conn(0x7f4b440cdae0 msgr2=0x7f4b440c83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 shutdown_connections 2024-04-25T11:44:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:15.664+0000 7f4b4bb07640 1 -- 172.21.15.136:0/3743049111 wait complete. 2024-04-25T11:44:15.732 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:44:17.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:18.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.164+0000 7f269f30c640 1 Processor -- start 2024-04-25T11:44:18.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.164+0000 7f269f30c640 1 -- start start 2024-04-25T11:44:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269f30c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2698152570 0x7f269806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269f30c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 0x7f2698162d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269f30c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26980c9760 con 0x7f269806d3b0 2024-04-25T11:44:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269f30c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2698158fd0 con 0x7f26980c6fa0 2024-04-25T11:44:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269f30c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f26981592b0 con 0x7f2698152570 2024-04-25T11:44:18.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2698152570 0x7f269806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2698152570 0x7f269806ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:56332/0 (socket says 172.21.15.136:56332) 2024-04-25T11:44:18.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 -- 172.21.15.136:0/4118092298 learned_addr learned my addr 172.21.15.136:0/4118092298 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:18.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 -- 172.21.15.136:0/4118092298 >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 msgr2=0x7f2698162d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269eb0b640 1 --2- 172.21.15.136:0/4118092298 >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 0x7f2698162d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 --2- 172.21.15.136:0/4118092298 >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 0x7f2698162d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 -- 172.21.15.136:0/4118092298 >> v1:172.21.15.136:6789/0 conn(0x7f269806d3b0 legacy=0x7f26980c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:18.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.168+0000 7f269e30a640 1 -- 172.21.15.136:0/4118092298 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269806c670 con 0x7f2698152570 2024-04-25T11:44:18.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.172+0000 7f269eb0b640 1 --2- 172.21.15.136:0/4118092298 >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 0x7f2698162d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:18.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.172+0000 7f269e30a640 1 --2- 172.21.15.136:0/4118092298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2698152570 0x7f269806ce70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f26980cdae0 tx=0x7f269401ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d971a09402b448 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.172+0000 7f269d308640 1 -- 172.21.15.136:0/4118092298 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26940b5070 con 0x7f2698152570 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.172+0000 7f269d308640 1 -- 172.21.15.136:0/4118092298 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26940a89b0 con 0x7f2698152570 2024-04-25T11:44:18.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269d308640 1 -- 172.21.15.136:0/4118092298 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26940a8d80 con 0x7f2698152570 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- 172.21.15.136:0/4118092298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2698152570 msgr2=0x7f269806ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 --2- 172.21.15.136:0/4118092298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2698152570 0x7f269806ce70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f26980cdae0 tx=0x7f269401ecd0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- 172.21.15.136:0/4118092298 shutdown_connections 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 --2- 172.21.15.136:0/4118092298 >> v2:172.21.15.136:3300/0 conn(0x7f26980c6fa0 0x7f2698162d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 --2- 172.21.15.136:0/4118092298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2698152570 0x7f269806ce70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- 172.21.15.136:0/4118092298 >> 172.21.15.136:0/4118092298 conn(0x7f26980cdb40 msgr2=0x7f26980cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- 172.21.15.136:0/4118092298 shutdown_connections 2024-04-25T11:44:18.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- 172.21.15.136:0/4118092298 wait complete. 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 Processor -- start 2024-04-25T11:44:18.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.176+0000 7f269f30c640 1 -- start start 2024-04-25T11:44:18.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f269806d3b0 0x7f2698151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2698152570 0x7f26981f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f26980ca270 con 0x7f2698152570 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f269806d7c0 con 0x7f26980c6fa0 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269f30c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f26980c73b0 con 0x7f269806d3b0 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:56344/0 (socket says 172.21.15.136:56344) 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 -- 172.21.15.136:0/1367417730 learned_addr learned my addr 172.21.15.136:0/1367417730 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 -- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f269806d3b0 msgr2=0x7f2698151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f269806d3b0 0x7f2698151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 -- 172.21.15.136:0/1367417730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2698152570 msgr2=0x7f26981f6c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2698152570 0x7f26981f6c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 -- 172.21.15.136:0/1367417730 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2698151b60 con 0x7f26980c6fa0 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.180+0000 7f269db09640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f268800b470 tx=0x7f268800b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688015030 con 0x7f26980c6fa0 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2688004020 con 0x7f26980c6fa0 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688004410 con 0x7f26980c6fa0 2024-04-25T11:44:18.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269814dbb0 con 0x7f26980c6fa0 2024-04-25T11:44:18.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2698150e20 con 0x7f26980c6fa0 2024-04-25T11:44:18.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.184+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269806a2e0 con 0x7f26980c6fa0 2024-04-25T11:44:18.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.188+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2688018070 con 0x7f26980c6fa0 2024-04-25T11:44:18.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.188+0000 7f268effd640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 0x7f267c0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:18.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.188+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7853+0+0 (secure 0 0 0) 0x7f268808f780 con 0x7f26980c6fa0 2024-04-25T11:44:18.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.192+0000 7f269e30a640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 0x7f267c0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:18.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.192+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f268800d040 con 0x7f26980c6fa0 2024-04-25T11:44:18.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.196+0000 7f269e30a640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 0x7f267c0719e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f26940b2000 tx=0x7f2694004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:18.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.760+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26980c7c40 con 0x7f26980c6fa0 2024-04-25T11:44:18.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.760+0000 7f268effd640 1 -- 172.21.15.136:0/1367417730 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6934 (secure 0 0 0) 0x7f2688062e20 con 0x7f26980c6fa0 2024-04-25T11:44:18.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:18.745 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5663,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:43:16.466037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5517,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5517":{"gid":5517,"name":"e","rank":2,"incarnation":38,"state":"up:active","state_seq":27,"addr":"172.21.15.136:6837/1740633057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1740633057},{"type":"v1","addr":"172.21.15.136:6837","nonce":1740633057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5517]},"id":1}]} 2024-04-25T11:44:18.746 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 97 2024-04-25T11:44:18.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.772+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 msgr2=0x7f267c0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.772+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 0x7f267c0719e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f26940b2000 tx=0x7f2694004740 comp rx=0 tx=0).stop 2024-04-25T11:44:18.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.772+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 msgr2=0x7f26980ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:18.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.772+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f268800b470 tx=0x7f268800b930 comp rx=0 tx=0).stop 2024-04-25T11:44:18.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 shutdown_connections 2024-04-25T11:44:18.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2698152570 0x7f26981f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26980c6fa0 0x7f26980ccc90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f267c06f4d0 0x7f267c0719e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 --2- 172.21.15.136:0/1367417730 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f269806d3b0 0x7f2698151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 >> 172.21.15.136:0/1367417730 conn(0x7f26980cdb40 msgr2=0x7f2698075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 shutdown_connections 2024-04-25T11:44:18.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:18.776+0000 7f269f30c640 1 -- 172.21.15.136:0/1367417730 wait complete. 2024-04-25T11:44:18.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-25T11:44:20.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:21.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 Processor -- start 2024-04-25T11:44:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 -- start start 2024-04-25T11:44:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181522f0 0x7fa918070700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa9180c9370 0x7fa9180c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa918152700 con 0x7fa91815b5d0 2024-04-25T11:44:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fa9180c8ee0 con 0x7fa9180c9370 2024-04-25T11:44:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.280+0000 7fa91c8cf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa9180729c0 con 0x7fa9181522f0 2024-04-25T11:44:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa916d9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa91815b5d0 0x7fa91815b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:43228/0 (socket says 172.21.15.136:43228) 2024-04-25T11:44:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa916d9d640 1 -- 172.21.15.136:0/334729442 learned_addr learned my addr 172.21.15.136:0/334729442 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:21.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa915d9b640 1 --2- 172.21.15.136:0/334729442 >> v2:172.21.15.136:3300/0 conn(0x7fa9180c9370 0x7fa9180c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:21.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa915d9b640 1 -- 172.21.15.136:0/334729442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181522f0 msgr2=0x7fa918070700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:21.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa915d9b640 1 --2- 172.21.15.136:0/334729442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181522f0 0x7fa918070700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa915d9b640 1 -- 172.21.15.136:0/334729442 >> v1:172.21.15.157:6789/0 conn(0x7fa91815b5d0 legacy=0x7fa91815b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.284+0000 7fa915d9b640 1 -- 172.21.15.136:0/334729442 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa918073f30 con 0x7fa9180c9370 2024-04-25T11:44:21.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa915d9b640 1 --2- 172.21.15.136:0/334729442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa9180c9780 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009a20 tx=0x7fa90c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6699f59474d8c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:21.272 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91559a640 1 -- 172.21.15.136:0/334729442 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0ab680 con 0x7fa9180c9370 2024-04-25T11:44:21.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91559a640 1 -- 172.21.15.136:0/334729442 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c0b0070 con 0x7fa9180c9370 2024-04-25T11:44:21.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91559a640 1 -- 172.21.15.136:0/334729442 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0aba40 con 0x7fa9180c9370 2024-04-25T11:44:21.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 -- 172.21.15.136:0/334729442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 msgr2=0x7fa9180c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 --2- 172.21.15.136:0/334729442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa9180c9780 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009a20 tx=0x7fa90c01c520 comp rx=0 tx=0).stop 2024-04-25T11:44:21.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 -- 172.21.15.136:0/334729442 shutdown_connections 2024-04-25T11:44:21.274 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 --2- 172.21.15.136:0/334729442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa9180c9780 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 --2- 172.21.15.136:0/334729442 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9181522f0 0x7fa918070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 -- 172.21.15.136:0/334729442 >> 172.21.15.136:0/334729442 conn(0x7fa9180c9d50 msgr2=0x7fa9180ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:21.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 -- 172.21.15.136:0/334729442 shutdown_connections 2024-04-25T11:44:21.275 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.288+0000 7fa91c8cf640 1 -- 172.21.15.136:0/334729442 wait complete. 2024-04-25T11:44:21.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.292+0000 7fa91c8cf640 1 Processor -- start 2024-04-25T11:44:21.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.292+0000 7fa91c8cf640 1 -- start start 2024-04-25T11:44:21.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.282 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa91815b5d0 0x7fa91815a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:56348/0 (socket says 172.21.15.136:56348) 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46276/0 (socket says 172.21.15.136:46276) 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa9180ca870 con 0x7fa9180c9370 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa9180c8b70 con 0x7fa9181522f0 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa9180ca9f0 con 0x7fa91815b5d0 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 -- 172.21.15.136:0/1366047871 learned_addr learned my addr 172.21.15.136:0/1366047871 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa916d9d640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa91815b5d0 0x7fa91815a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 -- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa91815b5d0 msgr2=0x7fa91815a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa91815b5d0 0x7fa91815a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 -- 172.21.15.136:0/1366047871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 msgr2=0x7fa918162960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 -- 172.21.15.136:0/1366047871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9181550b0 con 0x7fa9181522f0 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91659c640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa915d9b640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0ad040 tx=0x7fa90c0192c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c003bf0 con 0x7fa9181522f0 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 -- 172.21.15.136:0/1366047871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa918155390 con 0x7fa9181522f0 2024-04-25T11:44:21.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.296+0000 7fa91c8cf640 1 -- 172.21.15.136:0/1366047871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa918154820 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c0a7070 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0ab480 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8fc0052a0 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa90c0ab640 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa906ffd640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 0x7fa8f8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa91659c640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 0x7fa8f8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fa90c12a1b0 con 0x7fa9181522f0 2024-04-25T11:44:21.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.300+0000 7fa91659c640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 0x7fa8f8071940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa91000a850 tx=0x7fa91000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:21.297 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.308+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa90c016230 con 0x7fa9181522f0 2024-04-25T11:44:21.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.812+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8fc005590 con 0x7fa9181522f0 2024-04-25T11:44:21.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.812+0000 7fa906ffd640 1 -- 172.21.15.136:0/1366047871 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+6101 (secure 0 0 0) 0x7fa90c0f6250 con 0x7fa9181522f0 2024-04-25T11:44:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:21.799 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:19.077557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:21.799 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 98 2024-04-25T11:44:21.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.816+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 msgr2=0x7fa8f8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.816+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 0x7fa8f8071940 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa91000a850 tx=0x7fa91000d040 comp rx=0 tx=0).stop 2024-04-25T11:44:21.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.816+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 msgr2=0x7fa91814f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:21.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.816+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0ad040 tx=0x7fa90c0192c0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 shutdown_connections 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa8f806f430 0x7fa8f8071940 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa91815b5d0 0x7fa91815a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9181522f0 0x7fa91814f380 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 --2- 172.21.15.136:0/1366047871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9180c9370 0x7fa918162960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:21.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 >> 172.21.15.136:0/1366047871 conn(0x7fa9180c9d50 msgr2=0x7fa9180c6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 shutdown_connections 2024-04-25T11:44:21.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:21.820+0000 7fa904ff9640 1 -- 172.21.15.136:0/1366047871 wait complete. 2024-04-25T11:44:21.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-04-25T11:44:21.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-25T11:44:21.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:22.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.448+0000 7f8c81d12640 1 Processor -- start 2024-04-25T11:44:22.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c81d12640 1 -- start start 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c81d12640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c81d12640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c7c0c5fa0 con 0x7f8c7c142150 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c81d12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c7c158ff0 con 0x7f8c7c0c7b30 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c81d12640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f8c7c0c76a0 con 0x7f8c7c0c5650 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46290/0 (socket says 172.21.15.136:46290) 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 -- 172.21.15.136:0/3268637551 learned_addr learned my addr 172.21.15.136:0/3268637551 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 -- 172.21.15.136:0/3268637551 >> v1:172.21.15.157:6789/0 conn(0x7f8c7c0c7b30 legacy=0x7f8c7c0c7f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 -- 172.21.15.136:0/3268637551 >> v1:172.21.15.136:6789/0 conn(0x7f8c7c142150 legacy=0x7f8c7c070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:22.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.452+0000 7f8c7affd640 1 -- 172.21.15.136:0/3268637551 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c1592d0 con 0x7f8c7c0c5650 2024-04-25T11:44:22.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.468+0000 7f8c7affd640 1 --2- 172.21.15.136:0/3268637551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009870 tx=0x7f8c7001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e92305b652991a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:22.458 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.472+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/3268637551 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c700ab660 con 0x7f8c7c0c5650 2024-04-25T11:44:22.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.472+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/3268637551 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c700b0070 con 0x7f8c7c0c5650 2024-04-25T11:44:22.459 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.472+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/3268637551 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c700aba20 con 0x7f8c7c0c5650 2024-04-25T11:44:22.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.476+0000 7f8c81d12640 1 -- 172.21.15.136:0/3268637551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c0c5650 msgr2=0x7f8c7c0c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:22.460 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.476+0000 7f8c81d12640 1 --2- 172.21.15.136:0/3268637551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009870 tx=0x7f8c7001c460 comp rx=0 tx=0).stop 2024-04-25T11:44:22.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.480+0000 7f8c81d12640 1 -- 172.21.15.136:0/3268637551 shutdown_connections 2024-04-25T11:44:22.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.480+0000 7f8c81d12640 1 --2- 172.21.15.136:0/3268637551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c0c5a60 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.463 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.480+0000 7f8c81d12640 1 -- 172.21.15.136:0/3268637551 >> 172.21.15.136:0/3268637551 conn(0x7f8c7c06d780 msgr2=0x7f8c7c06db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:22.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.480+0000 7f8c81d12640 1 -- 172.21.15.136:0/3268637551 shutdown_connections 2024-04-25T11:44:22.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.480+0000 7f8c81d12640 1 -- 172.21.15.136:0/3268637551 wait complete. 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.484+0000 7f8c81d12640 1 Processor -- start 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.484+0000 7f8c81d12640 1 -- start start 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.484+0000 7f8c81d12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.484+0000 7f8c81d12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c7c0c7b30 0x7f8c7c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.484+0000 7f8c81d12640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46302/0 (socket says 172.21.15.136:46302) 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 -- 172.21.15.136:0/2901097312 learned_addr learned my addr 172.21.15.136:0/2901097312 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c159b70 con 0x7f8c7c142150 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c0c8f70 con 0x7f8c7c0c5650 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7c070b90 con 0x7f8c7c0c7b30 2024-04-25T11:44:22.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7b7fe640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 -- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c7c0c7b30 msgr2=0x7f8c7c15c830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c7c0c7b30 0x7f8c7c15c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 -- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 msgr2=0x7f8c7c154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c15cd70 con 0x7f8c7c142150 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7b7fe640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c7bfff640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400b960 tx=0x7f8c7400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74019070 con 0x7f8c7c142150 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7400cb20 con 0x7f8c7c142150 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74014440 con 0x7f8c7c142150 2024-04-25T11:44:22.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.488+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c15b230 con 0x7f8c7c142150 2024-04-25T11:44:22.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c7401c360 con 0x7f8c7c142150 2024-04-25T11:44:22.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c63fff640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 0x7f8c68071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:22.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c7b7fe640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 0x7f8c68071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:22.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c15c030 con 0x7f8c7c142150 2024-04-25T11:44:22.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c7b7fe640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 0x7f8c68071990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c001b60 tx=0x7f8c6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:22.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.492+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f8c74057710 con 0x7f8c7c142150 2024-04-25T11:44:22.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.496+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7c06a2e0 con 0x7f8c7c142150 2024-04-25T11:44:22.489 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.504+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8c7405b480 con 0x7f8c7c142150 2024-04-25T11:44:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.936+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c7c15af80 con 0x7f8c7c142150 2024-04-25T11:44:22.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.940+0000 7f8c63fff640 1 -- 172.21.15.136:0/2901097312 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+6101 (secure 0 0 0) 0x7f8c7405f780 con 0x7f8c7c142150 2024-04-25T11:44:22.926 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:22.926 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:19.077557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 98 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 msgr2=0x7f8c68071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 0x7f8c68071990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c001b60 tx=0x7f8c6c009040 comp rx=0 tx=0).stop 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 msgr2=0x7f8c7c155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400b960 tx=0x7f8c7400be20 comp rx=0 tx=0).stop 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 shutdown_connections 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c7c142150 0x7f8c7c155da0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6806f480 0x7f8c68071990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c7c0c7b30 0x7f8c7c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 --2- 172.21.15.136:0/2901097312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c7c0c5650 0x7f8c7c154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:22.932 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 >> 172.21.15.136:0/2901097312 conn(0x7f8c7c06d780 msgr2=0x7f8c7c0cc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 shutdown_connections 2024-04-25T11:44:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:22.948+0000 7f8c81d12640 1 -- 172.21.15.136:0/2901097312 wait complete. 2024-04-25T11:44:22.970 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 98, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5663": { "addr": "172.21.15.157:6839/3709442027", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3709442027, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3709442027, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5663, "incarnation": 98, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5977": { "addr": "172.21.15.157:6837/3388366693", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3388366693, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3388366693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5977, "incarnation": 92, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 16 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:44:19.077557+0000", "qdb_cluster": [ 5839, 5911, 5977, 6246 ], "qdb_leader": 5911, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5977, "mds_2": 5663, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:44:24.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:25.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 Processor -- start 2024-04-25T11:44:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 -- start start 2024-04-25T11:44:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2010070700 0x7f201015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2010152700 con 0x7f20101522f0 2024-04-25T11:44:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2010070b10 con 0x7f201015bb10 2024-04-25T11:44:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f2015aea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f20100c8ee0 con 0x7f2010070700 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200ffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f201015bb10 0x7f2010162960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:36138/0 (socket says 172.21.15.136:36138) 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200ffff640 1 -- 172.21.15.136:0/967560177 learned_addr learned my addr 172.21.15.136:0/967560177 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200effd640 1 --2- 172.21.15.136:0/967560177 >> v2:172.21.15.136:3300/0 conn(0x7f2010070700 0x7f201015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200e7fc640 1 -- 172.21.15.136:0/967560177 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 542612017 0 0) 0x7f2010070b10 con 0x7f201015bb10 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200e7fc640 1 -- 172.21.15.136:0/967560177 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ff8003530 con 0x7f201015bb10 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200effd640 1 -- 172.21.15.136:0/967560177 >> v1:172.21.15.157:6789/0 conn(0x7f201015bb10 legacy=0x7f2010162960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200effd640 1 -- 172.21.15.136:0/967560177 >> v1:172.21.15.136:6789/0 conn(0x7f20101522f0 legacy=0x7f20100c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:25.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.352+0000 7f200effd640 1 -- 172.21.15.136:0/967560177 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20100729c0 con 0x7f2010070700 2024-04-25T11:44:25.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.356+0000 7f200effd640 1 --2- 172.21.15.136:0/967560177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2010070700 0x7f201015b5d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f200000ba50 tx=0x7f200001eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d7ee16bd419f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:25.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.356+0000 7f200e7fc640 1 -- 172.21.15.136:0/967560177 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000ad070 con 0x7f2010070700 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.356+0000 7f200e7fc640 1 -- 172.21.15.136:0/967560177 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20000a7ac0 con 0x7f2010070700 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- 172.21.15.136:0/967560177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2010070700 msgr2=0x7f201015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f200e7fc640 1 -- 172.21.15.136:0/967560177 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000ad070 con 0x7f2010070700 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 --2- 172.21.15.136:0/967560177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2010070700 0x7f201015b5d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f200000ba50 tx=0x7f200001eb00 comp rx=0 tx=0).stop 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- 172.21.15.136:0/967560177 shutdown_connections 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 --2- 172.21.15.136:0/967560177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2010070700 0x7f201015b5d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- 172.21.15.136:0/967560177 >> 172.21.15.136:0/967560177 conn(0x7f20100c9d50 msgr2=0x7f20100ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- 172.21.15.136:0/967560177 shutdown_connections 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- 172.21.15.136:0/967560177 wait complete. 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 Processor -- start 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.360+0000 7f2015aea640 1 -- start start 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f201015bb10 0x7f2010157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44256/0 (socket says 172.21.15.136:44256) 2024-04-25T11:44:25.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 -- 172.21.15.136:0/2864133419 learned_addr learned my addr 172.21.15.136:0/2864133419 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:25.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2010214490 0x7f20101495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20100c9a80 con 0x7f201015bb10 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f20100c8b70 con 0x7f2010214490 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f20101668a0 con 0x7f20101522f0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200ffff640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2010214490 0x7f20101495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 -- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2010214490 msgr2=0x7f20101495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2010214490 0x7f20101495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 -- 172.21.15.136:0/2864133419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f201015bb10 msgr2=0x7f2010157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f201015bb10 0x7f2010157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201014d280 con 0x7f20101522f0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f200f7fe640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f200400b410 tx=0x7f200400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004015030 con 0x7f20101522f0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2004004020 con 0x7f20101522f0 2024-04-25T11:44:25.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004004450 con 0x7f20101522f0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2010155750 con 0x7f20101522f0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.364+0000 7f2015aea640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20102139b0 con 0x7f20101522f0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.368+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2004018070 con 0x7f20101522f0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.368+0000 7f1ff7fff640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 0x7f1ffc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.368+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f200408f5c0 con 0x7f20101522f0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.368+0000 7f200effd640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 0x7f1ffc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:25.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.368+0000 7f200effd640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 0x7f1ffc071940 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f20000b2000 tx=0x7f2000004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:25.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.380+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20102137a0 con 0x7f20101522f0 2024-04-25T11:44:25.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:25.392+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f200405bc00 con 0x7f20101522f0 2024-04-25T11:44:25.984 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.000+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201006a350 con 0x7f20101522f0 2024-04-25T11:44:25.986 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.000+0000 7f1ff7fff640 1 -- 172.21.15.136:0/2864133419 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+6103 (secure 0 0 0) 0x7f200405b500 con 0x7f20101522f0 2024-04-25T11:44:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:25.987 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:24.144676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:resolve","state_seq":17,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839]},"id":1}]} 2024-04-25T11:44:25.987 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 99 2024-04-25T11:44:25.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 msgr2=0x7f1ffc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 0x7f1ffc071940 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f20000b2000 tx=0x7f2000004740 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 msgr2=0x7f2010156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f200400b410 tx=0x7f200400b8d0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 shutdown_connections 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2010214490 0x7f20101495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f201015bb10 0x7f2010157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1ffc06f430 0x7f1ffc071940 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 --2- 172.21.15.136:0/2864133419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20101522f0 0x7f2010156d70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 >> 172.21.15.136:0/2864133419 conn(0x7f20100c9d50 msgr2=0x7f2010150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 shutdown_connections 2024-04-25T11:44:25.995 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:26.008+0000 7f1ff5ffb640 1 -- 172.21.15.136:0/2864133419 wait complete. 2024-04-25T11:44:26.046 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:44:28.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:28.322 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.336+0000 7fd689da8640 1 Processor -- start 2024-04-25T11:44:28.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.336+0000 7fd689da8640 1 -- start start 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd689da8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd67c0334f0 0x7fd67c006400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd689da8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd67c050ee0 0x7fd67c006940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd689da8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd67c0df820 con 0x7fd67c04ba50 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd689da8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd67c04c510 con 0x7fd67c050ee0 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd689da8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd67c0de930 con 0x7fd67c0334f0 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd683fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd67c0334f0 0x7fd67c006400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:28.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd683fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd67c0334f0 0x7fd67c006400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41234/0 (socket says 172.21.15.136:41234) 2024-04-25T11:44:28.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd683fff640 1 -- 172.21.15.136:0/3325630371 learned_addr learned my addr 172.21.15.136:0/3325630371 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:28.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.340+0000 7fd6895a7640 1 --2- 172.21.15.136:0/3325630371 >> v2:172.21.15.136:3300/0 conn(0x7fd67c050ee0 0x7fd67c006940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:28.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.344+0000 7fd683fff640 1 -- 172.21.15.136:0/3325630371 >> v2:172.21.15.136:3300/0 conn(0x7fd67c050ee0 msgr2=0x7fd67c006940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:28.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.344+0000 7fd683fff640 1 --2- 172.21.15.136:0/3325630371 >> v2:172.21.15.136:3300/0 conn(0x7fd67c050ee0 0x7fd67c006940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.344+0000 7fd683fff640 1 -- 172.21.15.136:0/3325630371 >> v1:172.21.15.136:6789/0 conn(0x7fd67c04ba50 legacy=0x7fd67c0df110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:28.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.344+0000 7fd683fff640 1 -- 172.21.15.136:0/3325630371 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c0351a0 con 0x7fd67c0334f0 2024-04-25T11:44:28.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.348+0000 7fd683fff640 1 --2- 172.21.15.136:0/3325630371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c0334f0 0x7fd67c006400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd67c051350 tx=0x7fd67002d2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=849856c1c8706bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.352+0000 7fd6837fe640 1 -- 172.21.15.136:0/3325630371 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700112d0 con 0x7fd67c0334f0 2024-04-25T11:44:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.352+0000 7fd6837fe640 1 -- 172.21.15.136:0/3325630371 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd670011470 con 0x7fd67c0334f0 2024-04-25T11:44:28.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.352+0000 7fd6837fe640 1 -- 172.21.15.136:0/3325630371 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67000f850 con 0x7fd67c0334f0 2024-04-25T11:44:28.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- 172.21.15.136:0/3325630371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c0334f0 msgr2=0x7fd67c006400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 --2- 172.21.15.136:0/3325630371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c0334f0 0x7fd67c006400 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd67c051350 tx=0x7fd67002d2d0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- 172.21.15.136:0/3325630371 shutdown_connections 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 --2- 172.21.15.136:0/3325630371 >> v2:172.21.15.136:3300/0 conn(0x7fd67c050ee0 0x7fd67c006940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 --2- 172.21.15.136:0/3325630371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c0334f0 0x7fd67c006400 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- 172.21.15.136:0/3325630371 >> 172.21.15.136:0/3325630371 conn(0x7fd67c0503c0 msgr2=0x7fd67c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- 172.21.15.136:0/3325630371 shutdown_connections 2024-04-25T11:44:28.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- 172.21.15.136:0/3325630371 wait complete. 2024-04-25T11:44:28.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 Processor -- start 2024-04-25T11:44:28.358 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.372+0000 7fd689da8640 1 -- start start 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45810/0 (socket says 172.21.15.136:45810) 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 -- 172.21.15.136:0/4109809162 learned_addr learned my addr 172.21.15.136:0/4109809162 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd67c04ba50 0x7fd67c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 0x7fd67c0d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c0debf0 con 0x7fd67c0334f0 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c035760 con 0x7fd67c050ee0 2024-04-25T11:44:28.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd67c034ea0 con 0x7fd67c04ba50 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd6895a7640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 0x7fd67c0d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 -- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd67c04ba50 msgr2=0x7fd67c0cbe70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd67c04ba50 0x7fd67c0cbe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 -- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 msgr2=0x7fd67c0d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 0x7fd67c0d00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c0d0e90 con 0x7fd67c0334f0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd6895a7640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 0x7fd67c0d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd688da6640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd678009e00 tx=0x7fd67800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678018030 con 0x7fd67c0334f0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67800cdb0 con 0x7fd67c0334f0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678004240 con 0x7fd67c0334f0 2024-04-25T11:44:28.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.376+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c166330 con 0x7fd67c0334f0 2024-04-25T11:44:28.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.380+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd678003600 con 0x7fd67c0334f0 2024-04-25T11:44:28.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.380+0000 7fd6817fa640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 0x7fd668071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:28.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.380+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c0cc710 con 0x7fd67c0334f0 2024-04-25T11:44:28.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.380+0000 7fd683fff640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 0x7fd668071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:28.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.380+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fd6780043e0 con 0x7fd67c0334f0 2024-04-25T11:44:28.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.384+0000 7fd683fff640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 0x7fd668071910 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd6700bf000 tx=0x7fd6700b8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:28.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.396+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c0cf1f0 con 0x7fd67c0334f0 2024-04-25T11:44:28.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.404+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd67805dd80 con 0x7fd67c0334f0 2024-04-25T11:44:28.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.952+0000 7fd689da8640 1 -- 172.21.15.136:0/4109809162 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd67c005580 con 0x7fd67c0334f0 2024-04-25T11:44:28.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.952+0000 7fd6817fa640 1 -- 172.21.15.136:0/4109809162 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+6103 (secure 0 0 0) 0x7fd67805d680 con 0x7fd67c0334f0 2024-04-25T11:44:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:28.941 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:24.144676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:resolve","state_seq":17,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839]},"id":1}]} 2024-04-25T11:44:28.941 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 99 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 msgr2=0x7fd668071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 0x7fd668071910 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd6700bf000 tx=0x7fd6700b8000 comp rx=0 tx=0).stop 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 msgr2=0x7fd67c166a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd678009e00 tx=0x7fd67800c430 comp rx=0 tx=0).stop 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 shutdown_connections 2024-04-25T11:44:28.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd67c050ee0 0x7fd67c0d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd66806f400 0x7fd668071910 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd67c04ba50 0x7fd67c0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 --2- 172.21.15.136:0/4109809162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd67c0334f0 0x7fd67c166a00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 >> 172.21.15.136:0/4109809162 conn(0x7fd67c0503c0 msgr2=0x7fd67c0ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 shutdown_connections 2024-04-25T11:44:28.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:28.960+0000 7fd66effd640 1 -- 172.21.15.136:0/4109809162 wait complete. 2024-04-25T11:44:28.985 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:44:30.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:31.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.232+0000 7facd7aa1640 1 Processor -- start 2024-04-25T11:44:31.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.232+0000 7facd7aa1640 1 -- start start 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd7aa1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd7aa1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7facd00ca720 0x7facd00ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd7aa1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd00cbad0 con 0x7facd00ce260 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd7aa1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7facd00c8550 con 0x7facd00ca720 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd7aa1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7facd00c7c20 con 0x7facd00c8cd0 2024-04-25T11:44:31.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd583d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd583d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41256/0 (socket says 172.21.15.136:41256) 2024-04-25T11:44:31.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd583d640 1 -- 172.21.15.136:0/1336631904 learned_addr learned my addr 172.21.15.136:0/1336631904 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:31.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd603e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7facd00ca720 0x7facd00ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd603e640 1 -- 172.21.15.136:0/1336631904 >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 msgr2=0x7facd014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd603e640 1 --2- 172.21.15.136:0/1336631904 >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd603e640 1 -- 172.21.15.136:0/1336631904 >> v1:172.21.15.136:6789/0 conn(0x7facd00ce260 legacy=0x7facd00ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd603e640 1 -- 172.21.15.136:0/1336631904 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd00c7f00 con 0x7facd00ca720 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.236+0000 7facd583d640 1 --2- 172.21.15.136:0/1336631904 >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.240+0000 7facd603e640 1 --2- 172.21.15.136:0/1336631904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00ca720 0x7facd00ce670 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7facd00c86d0 tx=0x7facc801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314161fab20a37bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.240+0000 7facd483b640 1 -- 172.21.15.136:0/1336631904 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc80a5b20 con 0x7facd00ca720 2024-04-25T11:44:31.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.240+0000 7facd483b640 1 -- 172.21.15.136:0/1336631904 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc80a5cc0 con 0x7facd00ca720 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd483b640 1 -- 172.21.15.136:0/1336631904 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc80ad660 con 0x7facd00ca720 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 -- 172.21.15.136:0/1336631904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00ca720 msgr2=0x7facd00ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1336631904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00ca720 0x7facd00ce670 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7facd00c86d0 tx=0x7facc801cab0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 -- 172.21.15.136:0/1336631904 shutdown_connections 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1336631904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00ca720 0x7facd00ce670 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1336631904 >> v2:172.21.15.157:3300/0 conn(0x7facd00c8cd0 0x7facd014f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.244+0000 7facd7aa1640 1 -- 172.21.15.136:0/1336631904 >> 172.21.15.136:0/1336631904 conn(0x7facd006f9d0 msgr2=0x7facd006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:31.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.248+0000 7facd7aa1640 1 -- 172.21.15.136:0/1336631904 shutdown_connections 2024-04-25T11:44:31.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.248+0000 7facd7aa1640 1 -- 172.21.15.136:0/1336631904 wait complete. 2024-04-25T11:44:31.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.248+0000 7facd7aa1640 1 Processor -- start 2024-04-25T11:44:31.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.248+0000 7facd7aa1640 1 -- start start 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 0x7facd0158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7facd00cb720 con 0x7facd00c8cd0 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7facd014fbc0 con 0x7facd00ca720 2024-04-25T11:44:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7facd01586d0 con 0x7facd00ce260 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd603e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd583d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 0x7facd0158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41262/0 (socket says 172.21.15.136:41262) 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd603e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44266/0 (socket says 172.21.15.136:44266) 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd583d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 0x7facd0158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:45824/0 (socket says 172.21.15.136:45824) 2024-04-25T11:44:31.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 -- 172.21.15.136:0/1577207254 learned_addr learned my addr 172.21.15.136:0/1577207254 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:31.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 -- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 msgr2=0x7facd0148a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 -- 172.21.15.136:0/1577207254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 msgr2=0x7facd0158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 0x7facd0158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 -- 172.21.15.136:0/1577207254 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd006d3a0 con 0x7facd00ca720 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd603e640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd503c640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faccc009d90 tx=0x7faccc00c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc018030 con 0x7facd00ca720 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd01590a0 con 0x7facd00ca720 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd0157040 con 0x7facd00ca720 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faccc00ccf0 con 0x7facd00ca720 2024-04-25T11:44:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.252+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd006a350 con 0x7facd00ca720 2024-04-25T11:44:31.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.256+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc004260 con 0x7facd00ca720 2024-04-25T11:44:31.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.256+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faccc01b070 con 0x7facd00ca720 2024-04-25T11:44:31.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.260+0000 7facc67fc640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 0x7facb40719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:31.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.260+0000 7facd583d640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 0x7facb40719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:31.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.260+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7faccc092840 con 0x7facd00ca720 2024-04-25T11:44:31.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.260+0000 7facd583d640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 0x7facb40719e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7facc0009650 tx=0x7facc0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:31.244 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.260+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facd006d3a0 con 0x7facd00ca720 2024-04-25T11:44:31.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.836+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facd00cc400 con 0x7facd00ca720 2024-04-25T11:44:31.823 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.840+0000 7facc67fc640 1 -- 172.21.15.136:0/1577207254 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6107 (secure 0 0 0) 0x7faccc05ee50 con 0x7facd00ca720 2024-04-25T11:44:31.824 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:31.824 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:29.489671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:31.824 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 100 2024-04-25T11:44:31.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 msgr2=0x7facb40719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 0x7facb40719e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7facc0009650 tx=0x7facc0009240 comp rx=0 tx=0).stop 2024-04-25T11:44:31.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 msgr2=0x7facd006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:31.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faccc009d90 tx=0x7faccc00c430 comp rx=0 tx=0).stop 2024-04-25T11:44:31.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 shutdown_connections 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7facb406f4d0 0x7facb40719e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7facd00ce260 0x7facd0148a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7facd00ca720 0x7facd006c840 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 --2- 172.21.15.136:0/1577207254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7facd00c8cd0 0x7facd0158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 >> 172.21.15.136:0/1577207254 conn(0x7facd006f9d0 msgr2=0x7facd00c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 shutdown_connections 2024-04-25T11:44:31.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:31.844+0000 7facd7aa1640 1 -- 172.21.15.136:0/1577207254 wait complete. 2024-04-25T11:44:31.869 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:44:33.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:34.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 Processor -- start 2024-04-25T11:44:34.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 -- start start 2024-04-25T11:44:34.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7eff640334f0 0x7eff64006940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff640df820 con 0x7eff6404c000 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7eff6404c510 con 0x7eff640334f0 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff72000640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7eff640de930 con 0x7eff6404ba50 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44282/0 (socket says 172.21.15.136:44282) 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 -- 172.21.15.136:0/1524236602 learned_addr learned my addr 172.21.15.136:0/1524236602 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 -- 172.21.15.136:0/1524236602 >> v2:172.21.15.157:3300/0 conn(0x7eff640334f0 msgr2=0x7eff64006940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 --2- 172.21.15.136:0/1524236602 >> v2:172.21.15.157:3300/0 conn(0x7eff640334f0 0x7eff64006940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 -- 172.21.15.136:0/1524236602 >> v1:172.21.15.157:6789/0 conn(0x7eff6404c000 legacy=0x7eff640df110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:44:34.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.080+0000 7eff6bfff640 1 -- 172.21.15.136:0/1524236602 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff640351a0 con 0x7eff6404ba50 2024-04-25T11:44:34.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.084+0000 7eff6bfff640 1 --2- 172.21.15.136:0/1524236602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000b70 tx=0x7eff6c0a2140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33aba4ba44e9c5b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:34.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.088+0000 7eff6b7fe640 1 -- 172.21.15.136:0/1524236602 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c12abd0 con 0x7eff6404ba50 2024-04-25T11:44:34.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.088+0000 7eff6b7fe640 1 -- 172.21.15.136:0/1524236602 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c12adb0 con 0x7eff6404ba50 2024-04-25T11:44:34.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.088+0000 7eff6b7fe640 1 -- 172.21.15.136:0/1524236602 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c132640 con 0x7eff6404ba50 2024-04-25T11:44:34.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.088+0000 7eff72000640 1 -- 172.21.15.136:0/1524236602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 msgr2=0x7eff64006400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:34.073 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.088+0000 7eff72000640 1 --2- 172.21.15.136:0/1524236602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000b70 tx=0x7eff6c0a2140 comp rx=0 tx=0).stop 2024-04-25T11:44:34.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.092+0000 7eff72000640 1 -- 172.21.15.136:0/1524236602 shutdown_connections 2024-04-25T11:44:34.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.092+0000 7eff72000640 1 --2- 172.21.15.136:0/1524236602 >> v2:172.21.15.157:3300/0 conn(0x7eff640334f0 0x7eff64006940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.092+0000 7eff72000640 1 --2- 172.21.15.136:0/1524236602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff64006400 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.092+0000 7eff72000640 1 -- 172.21.15.136:0/1524236602 >> 172.21.15.136:0/1524236602 conn(0x7eff640503c0 msgr2=0x7eff640507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.092+0000 7eff72000640 1 -- 172.21.15.136:0/1524236602 shutdown_connections 2024-04-25T11:44:34.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.100+0000 7eff72000640 1 -- 172.21.15.136:0/1524236602 wait complete. 2024-04-25T11:44:34.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.100+0000 7eff72000640 1 Processor -- start 2024-04-25T11:44:34.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.100+0000 7eff72000640 1 -- start start 2024-04-25T11:44:34.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 0x7eff640d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff640d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 0x7eff64178590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7eff6404ad40 con 0x7eff6404c000 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7eff64034ea0 con 0x7eff640334f0 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7eff6404c800 con 0x7eff6404ba50 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff70ffe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 0x7eff640d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff70ffe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 0x7eff640d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:41272/0 (socket says 172.21.15.136:41272) 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff70ffe640 1 -- 172.21.15.136:0/3197295344 learned_addr learned my addr 172.21.15.136:0/3197295344 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 0x7eff64178590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff6bfff640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff640d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 -- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 msgr2=0x7eff640d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff640d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 -- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 msgr2=0x7eff640d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 0x7eff640d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 -- 172.21.15.136:0/3197295344 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff640cdd40 con 0x7eff6404c000 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff717ff640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 0x7eff64178590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7eff6400c040 tx=0x7eff5800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:34.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58015020 con 0x7eff6404c000 2024-04-25T11:44:34.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff58011850 con 0x7eff6404c000 2024-04-25T11:44:34.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58011c40 con 0x7eff6404c000 2024-04-25T11:44:34.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff640d9020 con 0x7eff6404c000 2024-04-25T11:44:34.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641790f0 con 0x7eff6404c000 2024-04-25T11:44:34.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.104+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff640d5d30 con 0x7eff6404c000 2024-04-25T11:44:34.104 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.120+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7eff58018020 con 0x7eff6404c000 2024-04-25T11:44:34.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.120+0000 7eff697fa640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 0x7eff50071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:34.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.120+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7eff5808f790 con 0x7eff6404c000 2024-04-25T11:44:34.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.120+0000 7eff70ffe640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 0x7eff50071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:34.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.120+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7eff5801f020 con 0x7eff6404c000 2024-04-25T11:44:34.107 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.124+0000 7eff70ffe640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 0x7eff50071910 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff600099a0 tx=0x7eff600090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:34.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.660+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff640d56b0 con 0x7eff6404c000 2024-04-25T11:44:34.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.664+0000 7eff697fa640 1 -- 172.21.15.136:0/3197295344 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+6107 (secure 0 0 0) 0x7eff5805b760 con 0x7eff6404c000 2024-04-25T11:44:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:34.650 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:29.489671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:34.650 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 100 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 msgr2=0x7eff50071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 0x7eff50071910 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7eff600099a0 tx=0x7eff600090e0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 msgr2=0x7eff64178590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 0x7eff64178590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7eff6400c040 tx=0x7eff5800b870 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 shutdown_connections 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff6404c000 0x7eff64178590 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7eff5006f400 0x7eff50071910 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7eff6404ba50 0x7eff640d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 --2- 172.21.15.136:0/3197295344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7eff640334f0 0x7eff640d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 >> 172.21.15.136:0/3197295344 conn(0x7eff640503c0 msgr2=0x7eff640cef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:34.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 shutdown_connections 2024-04-25T11:44:34.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:34.668+0000 7eff72000640 1 -- 172.21.15.136:0/3197295344 wait complete. 2024-04-25T11:44:34.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:44:36.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:36.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 Processor -- start 2024-04-25T11:44:36.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 -- start start 2024-04-25T11:44:36.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f226c0738e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226c071300 con 0x7f226c0c4750 2024-04-25T11:44:36.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226c0c7300 con 0x7f226c152900 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2273ccb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f226c0710d0 con 0x7f226c0734d0 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2271a67640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f226c0c4750 0x7f226c158590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46190/0 (socket says 172.21.15.136:46190) 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2271a67640 1 -- 172.21.15.136:0/3302118755 learned_addr learned my addr 172.21.15.136:0/3302118755 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2272268640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f226c0738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769185695 0 0) 0x7f226c071300 con 0x7f226c0c4750 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2258003530 con 0x7f226c0c4750 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1010833964 0 0) 0x7f2258003530 con 0x7f226c0c4750 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 msgr2=0x7f226c0738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f226c0738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 >> v1:172.21.15.157:6789/0 conn(0x7f226c152900 legacy=0x7f226c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c0c8fa0 con 0x7f226c0c4750 2024-04-25T11:44:36.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f2268001680 con 0x7f226c0c4750 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 >> v1:172.21.15.136:6789/0 conn(0x7f226c0c4750 legacy=0x7f226c158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22580039f0 0x7f2258003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2258004810 0x7f2258004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f22580052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f225800bbb0 con 0x7f2258004810 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f225800bd60 con 0x7f22580039f0 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f225800bf60 con 0x7f226c0734d0 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2271266640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2258004810 0x7f2258004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2271a67640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22580039f0 0x7f2258003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.872+0000 7f2272268640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f22580052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 -- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 msgr2=0x7f22580052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:36.861 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f226c0734d0 0x7f22580052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 -- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22580039f0 msgr2=0x7f2258003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:36.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22580039f0 0x7f2258003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.862 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 -- 172.21.15.136:0/3302118755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2258003530 con 0x7f2258004810 2024-04-25T11:44:36.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2271266640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2258004810 0x7f2258004c20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f225c001860 tx=0x7f225c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c6a2f5e8cf6168 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:36.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c0b0070 con 0x7f2258004810 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225c0ab600 con 0x7f2258004810 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2270a65640 1 -- 172.21.15.136:0/3302118755 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c001460 con 0x7f2258004810 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 -- 172.21.15.136:0/3302118755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2258004810 msgr2=0x7f2258004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 --2- 172.21.15.136:0/3302118755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2258004810 0x7f2258004c20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f225c001860 tx=0x7f225c01c4e0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2272268640 1 -- 172.21.15.136:0/3302118755 reap_dead start 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 -- 172.21.15.136:0/3302118755 shutdown_connections 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 -- 172.21.15.136:0/3302118755 >> 172.21.15.136:0/3302118755 conn(0x7f226c0c9d50 msgr2=0x7f226c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 -- 172.21.15.136:0/3302118755 shutdown_connections 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.876+0000 7f2273ccb640 1 -- 172.21.15.136:0/3302118755 wait complete. 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 Processor -- start 2024-04-25T11:44:36.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- start start 2024-04-25T11:44:36.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2258004810 0x7f226c138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f226c162970 0x7f226c145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f226c152d10 con 0x7f226c1459c0 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f226c0c95f0 con 0x7f226c162970 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f226c0ca870 con 0x7f2258004810 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33306/0 (socket says 172.21.15.136:33306) 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 -- 172.21.15.136:0/615954890 learned_addr learned my addr 172.21.15.136:0/615954890 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2271a67640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2258004810 0x7f226c138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2271266640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f226c162970 0x7f226c145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 -- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2258004810 msgr2=0x7f226c138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2258004810 0x7f226c138b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 -- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f226c162970 msgr2=0x7f226c145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f226c162970 0x7f226c145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:36.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 -- 172.21.15.136:0/615954890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c156400 con 0x7f226c1459c0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2272268640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f226c06e190 tx=0x7f22640115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264011ea0 con 0x7f226c1459c0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f226401b070 con 0x7f226c1459c0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226c0ccf30 con 0x7f226c1459c0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22640039a0 con 0x7f226c1459c0 2024-04-25T11:44:36.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.880+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f226c0ccc50 con 0x7f226c1459c0 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f226c06a350 con 0x7f226c1459c0 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2264003b40 con 0x7f226c1459c0 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f22627fc640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 0x7f2250071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f2264091230 con 0x7f226c1459c0 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f2271a67640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 0x7f2250071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:36.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.884+0000 7f2271a67640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 0x7f2250071940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2268001210 tx=0x7f2268009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:36.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:36.888+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2264025020 con 0x7f226c1459c0 2024-04-25T11:44:37.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.084+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f226c071300 con 0x7f226c1459c0 2024-04-25T11:44:37.071 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f22627fc640 1 -- 172.21.15.136:0/615954890 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6104 (secure 0 0 0) 0x7f226405d980 con 0x7f226c1459c0 2024-04-25T11:44:37.072 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:37.072 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:35.030651+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839]},"id":1}]} 2024-04-25T11:44:37.072 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 101 2024-04-25T11:44:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 msgr2=0x7f2250071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 0x7f2250071940 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2268001210 tx=0x7f2268009040 comp rx=0 tx=0).stop 2024-04-25T11:44:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 msgr2=0x7f226c1610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f226c06e190 tx=0x7f22640115a0 comp rx=0 tx=0).stop 2024-04-25T11:44:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 shutdown_connections 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1459c0 0x7f226c1610c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f226c162970 0x7f226c145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f225006f430 0x7f2250071940 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 --2- 172.21.15.136:0/615954890 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2258004810 0x7f226c138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 >> 172.21.15.136:0/615954890 conn(0x7f226c0c9d50 msgr2=0x7f226c0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 shutdown_connections 2024-04-25T11:44:37.075 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:37.088+0000 7f2273ccb640 1 -- 172.21.15.136:0/615954890 wait complete. 2024-04-25T11:44:37.093 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T11:20:19.386747+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5663": { "addr": "172.21.15.157:6839/3709442027", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6838", "nonce": 3709442027, "type": "v2" }, { "addr": "172.21.15.157:6839", "nonce": 3709442027, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5663, "incarnation": 98, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:rejoin", "state_seq": 21 }, "gid_5839": { "addr": "172.21.15.157:6836/3569413002", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6834", "nonce": 3569413002, "type": "v2" }, { "addr": "172.21.15.157:6836", "nonce": 3569413002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5839, "incarnation": 80, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 19 }, "gid_5911": { "addr": "172.21.15.136:6833/2091616278", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6832", "nonce": 2091616278, "type": "v2" }, { "addr": "172.21.15.136:6833", "nonce": 2091616278, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5911, "incarnation": 86, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5977": { "addr": "172.21.15.157:6837/3388366693", "addrs": { "addrvec": [ { "addr": "172.21.15.157:6835", "nonce": 3388366693, "type": "v2" }, { "addr": "172.21.15.157:6837", "nonce": 3388366693, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5977, "incarnation": 92, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 16 }, "gid_6246": { "addr": "172.21.15.136:6835/4278341345", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 4278341345, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 4278341345, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6246, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T11:44:35.030651+0000", "qdb_cluster": [ 6246, 5977, 5911, 5839 ], "qdb_leader": 5911, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911, "mds_1": 5977, "mds_2": 5663, "mds_3": 5839, "mds_4": 6246 } } } ], "standbys": [] } 2024-04-25T11:44:39.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:39.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 Processor -- start 2024-04-25T11:44:39.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 -- start start 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0398159d20 0x7f0398153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03980ca460 con 0x7f03980cad00 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03980ca740 con 0x7f03980cc340 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f039d6c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f03980caa20 con 0x7f0398159d20 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f03977fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f03980cad00 0x7f03980cb110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46206/0 (socket says 172.21.15.136:46206) 2024-04-25T11:44:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f03977fe640 1 -- 172.21.15.136:0/3591926462 learned_addr learned my addr 172.21.15.136:0/3591926462 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:39.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f0396ffd640 1 --2- 172.21.15.136:0/3591926462 >> v2:172.21.15.136:3300/0 conn(0x7f0398159d20 0x7f0398153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:39.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f0396ffd640 1 -- 172.21.15.136:0/3591926462 >> v1:172.21.15.157:6789/0 conn(0x7f03980cc340 legacy=0x7f0398071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:39.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f0396ffd640 1 -- 172.21.15.136:0/3591926462 >> v1:172.21.15.136:6789/0 conn(0x7f03980cad00 legacy=0x7f03980cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f0395ffb640 1 -- 172.21.15.136:0/3591926462 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286171941 0 0) 0x7f03980ca460 con 0x7f03980cad00 2024-04-25T11:44:39.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.272+0000 7f0396ffd640 1 -- 172.21.15.136:0/3591926462 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03980cbb40 con 0x7f0398159d20 2024-04-25T11:44:39.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f0396ffd640 1 --2- 172.21.15.136:0/3591926462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f0398153fc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f03880097c0 tx=0x7f038801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850bb808f492219e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:39.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f0395ffb640 1 -- 172.21.15.136:0/3591926462 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880b0070 con 0x7f0398159d20 2024-04-25T11:44:39.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f0395ffb640 1 -- 172.21.15.136:0/3591926462 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03880ab600 con 0x7f0398159d20 2024-04-25T11:44:39.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f0395ffb640 1 -- 172.21.15.136:0/3591926462 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880ab9d0 con 0x7f0398159d20 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- 172.21.15.136:0/3591926462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 msgr2=0x7f0398153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 --2- 172.21.15.136:0/3591926462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f0398153fc0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f03880097c0 tx=0x7f038801c460 comp rx=0 tx=0).stop 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- 172.21.15.136:0/3591926462 shutdown_connections 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 --2- 172.21.15.136:0/3591926462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f0398153fc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- 172.21.15.136:0/3591926462 >> 172.21.15.136:0/3591926462 conn(0x7f0398068b90 msgr2=0x7f03980726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- 172.21.15.136:0/3591926462 shutdown_connections 2024-04-25T11:44:39.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- 172.21.15.136:0/3591926462 wait complete. 2024-04-25T11:44:39.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 Processor -- start 2024-04-25T11:44:39.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.276+0000 7f039d6c9640 1 -- start start 2024-04-25T11:44:39.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 0x7f0398146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:39.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 0x7f03981f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:39.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:39.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f03980cc8d0 con 0x7f0398159d20 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f03980cc750 con 0x7f03980cc340 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0398167ed0 con 0x7f03980cad00 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33318/0 (socket says 172.21.15.136:33318) 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 -- 172.21.15.136:0/1187798781 learned_addr learned my addr 172.21.15.136:0/1187798781 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f0396ffd640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 0x7f0398146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03967fc640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 0x7f03981f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 -- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 msgr2=0x7f0398146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 0x7f0398146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 -- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 msgr2=0x7f03981f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 0x7f03981f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 -- 172.21.15.136:0/1187798781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03981f3f00 con 0x7f0398159d20 2024-04-25T11:44:39.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03967fc640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 0x7f03981f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f0396ffd640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 0x7f0398146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f03977fe640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f039000c6a0 tx=0x7f039000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0390018070 con 0x7f0398159d20 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0390004d70 con 0x7f0398159d20 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03981f6900 con 0x7f0398159d20 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039000a880 con 0x7f0398159d20 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03981f7160 con 0x7f0398159d20 2024-04-25T11:44:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039815be70 con 0x7f0398159d20 2024-04-25T11:44:39.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.280+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f039001b070 con 0x7f0398159d20 2024-04-25T11:44:39.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.284+0000 7f037f7fe640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 0x7f0384071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:39.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.284+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f039008f060 con 0x7f0398159d20 2024-04-25T11:44:39.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.284+0000 7f0396ffd640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 0x7f0384071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:39.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.284+0000 7f0396ffd640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 0x7f0384071940 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:39.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.288+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f039005b720 con 0x7f0398159d20 2024-04-25T11:44:39.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03980b4840 con 0x7f0398159d20 2024-04-25T11:44:39.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f037f7fe640 1 -- 172.21.15.136:0/1187798781 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6104 (secure 0 0 0) 0x7f039005b020 con 0x7f0398159d20 2024-04-25T11:44:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:39.476 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:35.030651+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839]},"id":1}]} 2024-04-25T11:44:39.477 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 101 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 msgr2=0x7f0384071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 0x7f0384071940 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f03880ad000 tx=0x7f0388008040 comp rx=0 tx=0).stop 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 msgr2=0x7f03980cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.492+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f039000c6a0 tx=0x7f039000cb60 comp rx=0 tx=0).stop 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 shutdown_connections 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0398159d20 0x7f03980cead0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03980cc340 0x7f03981f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f038406f430 0x7f0384071940 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.479 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 --2- 172.21.15.136:0/1187798781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03980cad00 0x7f0398146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:39.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 >> 172.21.15.136:0/1187798781 conn(0x7f0398068b90 msgr2=0x7f039814c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:39.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 shutdown_connections 2024-04-25T11:44:39.480 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:39.496+0000 7f039d6c9640 1 -- 172.21.15.136:0/1187798781 wait complete. 2024-04-25T11:44:39.498 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:Syncing()...22 sec 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:FFSB benchmark finished at: Thu Apr 25 11:44:40 2024 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:Results: 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:Benchmark took 337.92 sec 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:Total Results 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout:=============== 2024-04-25T11:44:40.058 INFO:tasks.workunit.client.0.smithi136.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout: ======= ============ ========= ======= =========== ========== 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout: write : 654080 1935.59 100.000% 100.000% 7.56MB/sec 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout:- 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout:1935.59 Transactions per Second 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout:Throughput Results 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout:=================== 2024-04-25T11:44:40.059 INFO:tasks.workunit.client.0.smithi136.stdout:Write Throughput: 7.56MB/sec 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout:System Call Latency statistics in millisecs 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout:===== 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: Min Avg Max Total Calls 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: ======== ======== ======== ============ 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout:[ open] 0.288000 2195.354309 41701.359375 511 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[5] 100.000000 - 200.000000 : 14 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[6] 50.000000 - 100.000000 : 22 2024-04-25T11:44:40.063 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[7] 20.000000 - 50.000000 : 73 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[8] 10.000000 - 20.000000 : 126 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[9] 5.000000 - 10.000000 : 134 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[10] 2.000000 - 5.000000 : 84 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[11] 1.000000 - 2.000000 : 19 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[12] 0.500000 - 1.000000 : 5 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[13] 0.200000 - 0.500000 : 2 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout:[ write] 0.093000 13.526063 294.597992 654080 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T11:44:40.064 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[4] 200.000000 - 500.000000 : 118 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[5] 100.000000 - 200.000000 : 5357 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[6] 50.000000 - 100.000000 : 32530 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[7] 20.000000 - 50.000000 : 110312 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[8] 10.000000 - 20.000000 : 109231 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[9] 5.000000 - 10.000000 : 84704 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[10] 2.000000 - 5.000000 : 64028 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[11] 1.000000 - 2.000000 : 29744 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[12] 0.500000 - 1.000000 : 42543 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[13] 0.200000 - 0.500000 : 135406 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[14] 0.100000 - 0.200000 : 39768 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[15] 0.050000 - 0.100000 : 339 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T11:44:40.065 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout:[ lseek] 0.000000 0.001158 6.291000 654080 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[10] 2.000000 - 5.000000 : 5 2024-04-25T11:44:40.066 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[11] 1.000000 - 2.000000 : 6 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[12] 0.500000 - 1.000000 : 7 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[13] 0.200000 - 0.500000 : 37 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[14] 0.100000 - 0.200000 : 33 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[15] 0.050000 - 0.100000 : 47 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[16] 0.020000 - 0.050000 : 304 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[17] 0.010000 - 0.020000 : 466 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[18] 0.000000 - 0.010000 : 653174 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout:[ close] 0.021000 4.259344 102.834999 511 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T11:44:40.067 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[6] 50.000000 - 100.000000 : 14 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[7] 20.000000 - 50.000000 : 19 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[8] 10.000000 - 20.000000 : 12 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[9] 5.000000 - 10.000000 : 15 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[10] 2.000000 - 5.000000 : 47 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[11] 1.000000 - 2.000000 : 35 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[12] 0.500000 - 1.000000 : 28 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[13] 0.200000 - 0.500000 : 74 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[14] 0.100000 - 0.200000 : 111 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[15] 0.050000 - 0.100000 : 93 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[16] 0.020000 - 0.050000 : 62 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-04-25T11:44:40.069 INFO:tasks.workunit.client.0.smithi136.stdout:0.0% User Time 2024-04-25T11:44:40.069 INFO:tasks.workunit.client.0.smithi136.stdout:42.7% System Time 2024-04-25T11:44:40.069 INFO:tasks.workunit.client.0.smithi136.stdout:42.7% CPU Utilization 2024-04-25T11:44:40.069 INFO:tasks.workunit.client.0.smithi136.stderr:+ cd .. 2024-04-25T11:44:40.079 INFO:tasks.workunit.client.0.smithi136.stderr:+ rm -r tmp ffsb 2024-04-25T11:44:41.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:41.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 Processor -- start 2024-04-25T11:44:41.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 -- start start 2024-04-25T11:44:41.686 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51080c96e0 0x7f51081540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51080c79e0 0x7f51080c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f510815a3d0 0x7f510815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f510806cc10 con 0x7f51080c96e0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f51080c9af0 con 0x7f510815a3d0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510f54c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51080cb220 con 0x7f51080c79e0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510d2e8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51080c96e0 0x7f51081540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510d2e8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51080c96e0 0x7f51081540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33324/0 (socket says 172.21.15.136:33324) 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510d2e8640 1 -- 172.21.15.136:0/1056543410 learned_addr learned my addr 172.21.15.136:0/1056543410 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510dae9640 1 --2- 172.21.15.136:0/1056543410 >> v2:172.21.15.157:3300/0 conn(0x7f510815a3d0 0x7f510815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.700+0000 7f510cae7640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51080c79e0 0x7f51080c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 -- 172.21.15.136:0/1056543410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51080c79e0 msgr2=0x7f51080c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:41.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51080c79e0 0x7f51080c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 -- 172.21.15.136:0/1056543410 >> v2:172.21.15.157:3300/0 conn(0x7f510815a3d0 msgr2=0x7f510815a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:41.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1056543410 >> v2:172.21.15.157:3300/0 conn(0x7f510815a3d0 0x7f510815a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.688 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 -- 172.21.15.136:0/1056543410 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51080cb500 con 0x7f51080c96e0 2024-04-25T11:44:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c96e0 0x7f51081540d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f51080729b0 tx=0x7f510001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fe09317826ce59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f50fffff640 1 -- 172.21.15.136:0/1056543410 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51000b4070 con 0x7f51080c96e0 2024-04-25T11:44:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f50fffff640 1 -- 172.21.15.136:0/1056543410 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51000af400 con 0x7f51080c96e0 2024-04-25T11:44:41.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f50fffff640 1 -- 172.21.15.136:0/1056543410 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51000af7d0 con 0x7f51080c96e0 2024-04-25T11:44:41.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 -- 172.21.15.136:0/1056543410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c96e0 msgr2=0x7f51081540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:41.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c96e0 0x7f51081540d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f51080729b0 tx=0x7f510001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 -- 172.21.15.136:0/1056543410 shutdown_connections 2024-04-25T11:44:41.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 --2- 172.21.15.136:0/1056543410 >> v2:172.21.15.157:3300/0 conn(0x7f510815a3d0 0x7f510815a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51080c79e0 0x7f51080c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 --2- 172.21.15.136:0/1056543410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c96e0 0x7f51081540d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 -- 172.21.15.136:0/1056543410 >> 172.21.15.136:0/1056543410 conn(0x7f510806f9d0 msgr2=0x7f510806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 -- 172.21.15.136:0/1056543410 shutdown_connections 2024-04-25T11:44:41.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.704+0000 7f510f54c640 1 -- 172.21.15.136:0/1056543410 wait complete. 2024-04-25T11:44:41.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 Processor -- start 2024-04-25T11:44:41.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 -- start start 2024-04-25T11:44:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 0x7f5108156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f510815a3d0 0x7f5108154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5108154610 con 0x7f51080c79e0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f510815ad20 con 0x7f51080c96e0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510f54c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51080cf0f0 con 0x7f510815a3d0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510d2e8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510d2e8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33336/0 (socket says 172.21.15.136:33336) 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.708+0000 7f510d2e8640 1 -- 172.21.15.136:0/1987799332 learned_addr learned my addr 172.21.15.136:0/1987799332 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510cae7640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 0x7f5108156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510dae9640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f510815a3d0 0x7f5108154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 -- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f510815a3d0 msgr2=0x7f5108154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f510815a3d0 0x7f5108154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 -- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 msgr2=0x7f5108156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 0x7f5108156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 -- 172.21.15.136:0/1987799332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51081e3470 con 0x7f51080c79e0 2024-04-25T11:44:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510cae7640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 0x7f5108156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510d2e8640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f510001d890 tx=0x7f510001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51000b4040 con 0x7f51080c79e0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51000c0070 con 0x7f51080c79e0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51081e4140 con 0x7f51080c79e0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100009510 con 0x7f51080c79e0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51081e5d30 con 0x7f51080c79e0 2024-04-25T11:44:41.697 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f510806a350 con 0x7f51080c79e0 2024-04-25T11:44:41.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51000ba660 con 0x7f51080c79e0 2024-04-25T11:44:41.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.712+0000 7f50fdffb640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 0x7f50ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:41.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.716+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f510012de10 con 0x7f51080c79e0 2024-04-25T11:44:41.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.716+0000 7f510cae7640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 0x7f50ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:41.699 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.716+0000 7f510cae7640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 0x7f50ec071910 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f51080c9330 tx=0x7f5104008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:41.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.724+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5100018d40 con 0x7f51080c79e0 2024-04-25T11:44:41.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.956+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51081579a0 con 0x7f51080c79e0 2024-04-25T11:44:41.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f50fdffb640 1 -- 172.21.15.136:0/1987799332 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6109 (secure 0 0 0) 0x7f51000fa590 con 0x7f51080c79e0 2024-04-25T11:44:41.943 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:41.943 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:41.473400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5663,5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:41.943 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 102 2024-04-25T11:44:41.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 msgr2=0x7f50ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:41.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 0x7f50ec071910 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f51080c9330 tx=0x7f5104008040 comp rx=0 tx=0).stop 2024-04-25T11:44:41.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 msgr2=0x7f51081e3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:41.945 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f510001d890 tx=0x7f510001bf80 comp rx=0 tx=0).stop 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 shutdown_connections 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f50ec06f400 0x7f50ec071910 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f510815a3d0 0x7f5108154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51080c96e0 0x7f5108156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 --2- 172.21.15.136:0/1987799332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51080c79e0 0x7f51081e3c00 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 >> 172.21.15.136:0/1987799332 conn(0x7f510806f9d0 msgr2=0x7f5108148cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 shutdown_connections 2024-04-25T11:44:41.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:41.960+0000 7f510f54c640 1 -- 172.21.15.136:0/1987799332 wait complete. 2024-04-25T11:44:41.973 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=5517 2024-04-25T11:44:41.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.e 2024-04-25T11:44:41.973 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-04-25T11:44:41.973 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-25T11:44:41.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-25T11:44:41.975 INFO:tasks.ceph.mds.e:Started 2024-04-25T11:44:41.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:42.081 INFO:tasks.ceph.mds.e.smithi136.stdout:starting mds.e at 2024-04-25T11:44:42.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 Processor -- start 2024-04-25T11:44:42.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 -- start start 2024-04-25T11:44:42.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f72980c79e0 0x7f72980c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f729815a3d0 0x7f729815a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72980c9af0 con 0x7f72980c96e0 2024-04-25T11:44:42.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f729806cc10 con 0x7f729815a3d0 2024-04-25T11:44:42.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729ebd6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f72980cb220 con 0x7f72980c79e0 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729d173640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f729815a3d0 0x7f729815a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729d173640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f729815a3d0 0x7f729815a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33342/0 (socket says 172.21.15.136:33342) 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729d173640 1 -- 172.21.15.136:0/2358457809 learned_addr learned my addr 172.21.15.136:0/2358457809 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.160+0000 7f729c972640 1 --1- 172.21.15.136:0/2358457809 >> v1:172.21.15.136:6789/0 conn(0x7f72980c96e0 0x7f72981540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46212/0 (socket says 172.21.15.136:46212) 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f7297fff640 1 --2- 172.21.15.136:0/2358457809 >> v2:172.21.15.157:3300/0 conn(0x7f72980c79e0 0x7f72980c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729d173640 1 -- 172.21.15.136:0/2358457809 >> v2:172.21.15.157:3300/0 conn(0x7f72980c79e0 msgr2=0x7f72980c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729d173640 1 --2- 172.21.15.136:0/2358457809 >> v2:172.21.15.157:3300/0 conn(0x7f72980c79e0 0x7f72980c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729d173640 1 -- 172.21.15.136:0/2358457809 >> v1:172.21.15.136:6789/0 conn(0x7f72980c96e0 legacy=0x7f72981540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729d173640 1 -- 172.21.15.136:0/2358457809 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72980cb500 con 0x7f729815a3d0 2024-04-25T11:44:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f72977fe640 1 -- 172.21.15.136:0/2358457809 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722892239 0 0) 0x7f72980c9af0 con 0x7f72980c96e0 2024-04-25T11:44:42.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729d173640 1 --2- 172.21.15.136:0/2358457809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f729815a3d0 0x7f729815a7e0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f72980c8990 tx=0x7f729001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb77f40772902456 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:42.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f72977fe640 1 -- 172.21.15.136:0/2358457809 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72900a56c0 con 0x7f729815a3d0 2024-04-25T11:44:42.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f72977fe640 1 -- 172.21.15.136:0/2358457809 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72900a5860 con 0x7f729815a3d0 2024-04-25T11:44:42.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f72977fe640 1 -- 172.21.15.136:0/2358457809 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72900a5c30 con 0x7f729815a3d0 2024-04-25T11:44:42.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 -- 172.21.15.136:0/2358457809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f729815a3d0 msgr2=0x7f729815a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2358457809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f729815a3d0 0x7f729815a7e0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f72980c8990 tx=0x7f729001c9f0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 -- 172.21.15.136:0/2358457809 shutdown_connections 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2358457809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f729815a3d0 0x7f729815a7e0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2358457809 >> v2:172.21.15.157:3300/0 conn(0x7f72980c79e0 0x7f72980c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.164+0000 7f729ebd6640 1 -- 172.21.15.136:0/2358457809 >> 172.21.15.136:0/2358457809 conn(0x7f729806f9d0 msgr2=0x7f729806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- 172.21.15.136:0/2358457809 shutdown_connections 2024-04-25T11:44:42.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- 172.21.15.136:0/2358457809 wait complete. 2024-04-25T11:44:42.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 Processor -- start 2024-04-25T11:44:42.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- start start 2024-04-25T11:44:42.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 0x7f729814a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f729815a3d0 0x7f729814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f72980ced40 con 0x7f72980c96e0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f729806d440 con 0x7f729815a3d0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729ebd6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72981f5320 con 0x7f72980c79e0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f7297fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f7297fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33344/0 (socket says 172.21.15.136:33344) 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f7297fff640 1 -- 172.21.15.136:0/2403474872 learned_addr learned my addr 172.21.15.136:0/2403474872 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729c972640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 0x7f729814a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.168+0000 7f729d173640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f729815a3d0 0x7f729814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 -- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 msgr2=0x7f729814a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 0x7f729814a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 -- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f729815a3d0 msgr2=0x7f729814e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f729815a3d0 0x7f729814e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 -- 172.21.15.136:0/2403474872 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72981598f0 con 0x7f72980c96e0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f729c972640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 0x7f729814a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f7297fff640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f728400d620 tx=0x7f728400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728400b620 con 0x7f72980c96e0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284019070 con 0x7f72980c96e0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7298159bd0 con 0x7f72980c96e0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728400b9e0 con 0x7f72980c96e0 2024-04-25T11:44:42.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72980cd680 con 0x7f72980c96e0 2024-04-25T11:44:42.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729806a350 con 0x7f72980c96e0 2024-04-25T11:44:42.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f728400bb80 con 0x7f72980c96e0 2024-04-25T11:44:42.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 0x7f727c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:42.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.172+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f728408feb0 con 0x7f72980c96e0 2024-04-25T11:44:42.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.176+0000 7f729c972640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 0x7f727c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:42.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.176+0000 7f729c972640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 0x7f727c071940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f728c000f70 tx=0x7f728c007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:42.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.180+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f728405c600 con 0x7f72980c96e0 2024-04-25T11:44:42.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.400+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72980cb860 con 0x7f72980c96e0 2024-04-25T11:44:42.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.400+0000 7f72957fa640 1 -- 172.21.15.136:0/2403474872 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6109 (secure 0 0 0) 0x7f728405bf00 con 0x7f72980c96e0 2024-04-25T11:44:42.387 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:42.387 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:41.473400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5663,5839,5911,5977,6246]},"id":1}]} 2024-04-25T11:44:42.387 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 102 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 msgr2=0x7f727c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 0x7f727c071940 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f728c000f70 tx=0x7f728c007cb0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 msgr2=0x7f72980cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f728400d620 tx=0x7f728400dae0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 shutdown_connections 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f729815a3d0 0x7f729814e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72980c96e0 0x7f72980cdba0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f727c06f430 0x7f727c071940 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 --2- 172.21.15.136:0/2403474872 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f72980c79e0 0x7f729814a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:42.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 >> 172.21.15.136:0/2403474872 conn(0x7f729806f9d0 msgr2=0x7f7298153330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:42.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 shutdown_connections 2024-04-25T11:44:42.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:42.404+0000 7f729ebd6640 1 -- 172.21.15.136:0/2403474872 wait complete. 2024-04-25T11:44:42.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-04-25T11:44:44.053 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T11:44:44.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T11:44:44.112 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-25T11:44:44.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T11:44:44.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:44:44.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 Processor -- start 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 -- start start 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 0x7f51e80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51e80074c0 0x7f51e8107850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e8045770 con 0x7f51e80070b0 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f51e80060a0 con 0x7f51e80074c0 2024-04-25T11:44:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f68bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f51e8006c20 con 0x7f51e8045360 2024-04-25T11:44:44.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f50bc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f51e80070b0 0x7f51e804bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39610/0 (socket says 172.21.15.136:39610) 2024-04-25T11:44:44.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f50bc640 1 -- 172.21.15.136:0/2043130783 learned_addr learned my addr 172.21.15.136:0/2043130783 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:44.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f60be640 1 --2- 172.21.15.136:0/2043130783 >> v2:172.21.15.136:3300/0 conn(0x7f51e80074c0 0x7f51e8107850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:44.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.884+0000 7f51f60be640 1 -- 172.21.15.136:0/2043130783 >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 msgr2=0x7f51e80f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f48bb640 1 -- 172.21.15.136:0/2043130783 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3088610885 0 0) 0x7f51e8045770 con 0x7f51e80070b0 2024-04-25T11:44:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.876+0000 7f51f58bd640 1 --2- 172.21.15.136:0/2043130783 >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 0x7f51e80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.888+0000 7f51f60be640 1 --2- 172.21.15.136:0/2043130783 >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 0x7f51e80f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.888+0000 7f51f60be640 1 -- 172.21.15.136:0/2043130783 >> v1:172.21.15.136:6789/0 conn(0x7f51e80070b0 legacy=0x7f51e804bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.888+0000 7f51f60be640 1 -- 172.21.15.136:0/2043130783 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e80078d0 con 0x7f51e80074c0 2024-04-25T11:44:44.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.892+0000 7f51f58bd640 1 --2- 172.21.15.136:0/2043130783 >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 0x7f51e80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:44.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.896+0000 7f51f60be640 1 --2- 172.21.15.136:0/2043130783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e8107850 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f51f00515a0 tx=0x7f51f008d440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52464abbc0c88d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:44.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.896+0000 7f51f48bb640 1 -- 172.21.15.136:0/2043130783 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0121070 con 0x7f51e80074c0 2024-04-25T11:44:44.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.896+0000 7f51f48bb640 1 -- 172.21.15.136:0/2043130783 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f011c440 con 0x7f51e80074c0 2024-04-25T11:44:44.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.896+0000 7f51f48bb640 1 -- 172.21.15.136:0/2043130783 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0072040 con 0x7f51e80074c0 2024-04-25T11:44:44.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.900+0000 7f51f68bf640 1 -- 172.21.15.136:0/2043130783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 msgr2=0x7f51e8107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:44.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.900+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2043130783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e8107850 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f51f00515a0 tx=0x7f51f008d440 comp rx=0 tx=0).stop 2024-04-25T11:44:44.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.904+0000 7f51f68bf640 1 -- 172.21.15.136:0/2043130783 shutdown_connections 2024-04-25T11:44:44.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.904+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2043130783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e8107850 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:44.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.904+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2043130783 >> v2:172.21.15.157:3300/0 conn(0x7f51e8045360 0x7f51e80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:44.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.904+0000 7f51f68bf640 1 -- 172.21.15.136:0/2043130783 >> 172.21.15.136:0/2043130783 conn(0x7f51e80503c0 msgr2=0x7f51e80507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:44.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.912+0000 7f51f68bf640 1 -- 172.21.15.136:0/2043130783 shutdown_connections 2024-04-25T11:44:44.895 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.912+0000 7f51f68bf640 1 -- 172.21.15.136:0/2043130783 wait complete. 2024-04-25T11:44:44.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 Processor -- start 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 -- start start 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51e80070b0 0x7f51e80f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51e8045360 0x7f51e8047e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51e80062f0 con 0x7f51e80074c0 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51e80fdcb0 con 0x7f51e80070b0 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f68bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51e80065d0 con 0x7f51e8045360 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f50bc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f50bc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33892/0 (socket says 172.21.15.136:33892) 2024-04-25T11:44:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.916+0000 7f51f50bc640 1 -- 172.21.15.136:0/2665511859 learned_addr learned my addr 172.21.15.136:0/2665511859 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 -- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51e8045360 msgr2=0x7f51e8047e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51e8045360 0x7f51e8047e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 -- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51e80070b0 msgr2=0x7f51e80f0d00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51e80070b0 0x7f51e80f0d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 -- 172.21.15.136:0/2665511859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e80f1240 con 0x7f51e80074c0 2024-04-25T11:44:44.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.920+0000 7f51f50bc640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c410 tx=0x7f51e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:44.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.924+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00074e0 con 0x7f51e80074c0 2024-04-25T11:44:44.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.924+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e00047f0 con 0x7f51e80074c0 2024-04-25T11:44:44.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.924+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0007680 con 0x7f51e80074c0 2024-04-25T11:44:44.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.924+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e80f3030 con 0x7f51e80074c0 2024-04-25T11:44:44.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.924+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51e0011070 con 0x7f51e80074c0 2024-04-25T11:44:44.910 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51e67fc640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 0x7f51d4071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:44.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e80f35f0 con 0x7f51e80074c0 2024-04-25T11:44:44.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51f58bd640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 0x7f51d4071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:44.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f51e0057090 con 0x7f51e80074c0 2024-04-25T11:44:44.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51f58bd640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 0x7f51d4071960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00a830 tx=0x7f51ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:44.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.928+0000 7f51d3fff640 1 -- 172.21.15.136:0/2665511859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e80f39d0 con 0x7f51e80074c0 2024-04-25T11:44:44.923 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:44.940+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f51e005b500 con 0x7f51e80074c0 2024-04-25T11:44:44.937 DEBUG:teuthology.parallel:result is None 2024-04-25T11:44:44.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T11:44:45.025 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T11:44:45.025 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-04-25T11:44:45.040 INFO:tasks.mds_thrash:joining mds_thrasher 2024-04-25T11:44:45.515 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.532+0000 7f51d3fff640 1 -- 172.21.15.136:0/2665511859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51e80f2e20 con 0x7f51e80074c0 2024-04-25T11:44:45.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.532+0000 7f51e67fc640 1 -- 172.21.15.136:0/2665511859 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+6937 (secure 0 0 0) 0x7f51e005ae00 con 0x7f51e80074c0 2024-04-25T11:44:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:45.516 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6852,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2988117878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2988117878},{"type":"v1","addr":"172.21.15.136:6837","nonce":2988117878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:41.473400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5663]},"id":1}]} 2024-04-25T11:44:45.516 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 103 2024-04-25T11:44:45.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.536+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 msgr2=0x7f51d4071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:45.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.536+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 0x7f51d4071960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00a830 tx=0x7f51ec008040 comp rx=0 tx=0).stop 2024-04-25T11:44:45.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.536+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 msgr2=0x7f51e804fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:45.523 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.536+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f51e000c410 tx=0x7f51e000c8d0 comp rx=0 tx=0).stop 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 shutdown_connections 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f51d406f450 0x7f51d4071960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51e8045360 0x7f51e8047e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e80074c0 0x7f51e804fa00 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 --2- 172.21.15.136:0/2665511859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51e80070b0 0x7f51e80f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:45.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 >> 172.21.15.136:0/2665511859 conn(0x7f51e80503c0 msgr2=0x7f51e80fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:45.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 shutdown_connections 2024-04-25T11:44:45.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:45.544+0000 7f51f68bf640 1 -- 172.21.15.136:0/2665511859 wait complete. 2024-04-25T11:44:45.565 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-04-25T11:44:45.565 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2024-04-25T11:44:45.565 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2024-04-25T11:44:45.565 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 14 2024-04-25T11:44:45.565 INFO:tasks.mds_thrash:done joining 2024-04-25T11:44:45.565 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T11:44:45.574 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T11:44:45.574 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:44:45.574 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:44:45.738 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:44:45.738 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:44:45.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:44:46.211 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.228+0000 7fe7f641b640 1 Processor -- start 2024-04-25T11:44:46.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.232+0000 7fe7f641b640 1 -- start start 2024-04-25T11:44:46.220 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7f641b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe7f00c2e00 0x7fe7f015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7f641b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7f641b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f00c3210 con 0x7fe7f0074ee0 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7f641b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe7f015cae0 con 0x7fe7f00c2e00 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50084/0 (socket says 172.21.15.136:50084) 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 -- 172.21.15.136:0/2013884000 learned_addr learned my addr 172.21.15.136:0/2013884000 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:46.221 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f00c9d20 con 0x7fe7f015c300 2024-04-25T11:44:46.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 -- 172.21.15.136:0/2013884000 >> v2:172.21.15.157:3300/0 conn(0x7fe7f00c2e00 msgr2=0x7fe7f015bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:46.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/2013884000 >> v2:172.21.15.157:3300/0 conn(0x7fe7f00c2e00 0x7fe7f015bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 -- 172.21.15.136:0/2013884000 >> v1:172.21.15.136:6789/0 conn(0x7fe7f0074ee0 legacy=0x7fe7f00c4340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:46.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.236+0000 7fe7ef7fe640 1 -- 172.21.15.136:0/2013884000 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f015c800 con 0x7fe7f015c300 2024-04-25T11:44:46.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.240+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/2013884000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00cb440 tx=0x7fe7e401e620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6878eb461390a38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:46.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.240+0000 7fe7eeffd640 1 -- 172.21.15.136:0/2013884000 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40b2070 con 0x7fe7f015c300 2024-04-25T11:44:46.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.240+0000 7fe7eeffd640 1 -- 172.21.15.136:0/2013884000 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e40ad600 con 0x7fe7f015c300 2024-04-25T11:44:46.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 msgr2=0x7fe7f00749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:46.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 --2- 172.21.15.136:0/2013884000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00cb440 tx=0x7fe7e401e620 comp rx=0 tx=0).stop 2024-04-25T11:44:46.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 shutdown_connections 2024-04-25T11:44:46.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 --2- 172.21.15.136:0/2013884000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f015c300 0x7fe7f00749a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 --2- 172.21.15.136:0/2013884000 >> v2:172.21.15.157:3300/0 conn(0x7fe7f00c2e00 0x7fe7f015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.244+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 >> 172.21.15.136:0/2013884000 conn(0x7fe7f00ca5a0 msgr2=0x7fe7f00ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:46.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.248+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 shutdown_connections 2024-04-25T11:44:46.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.248+0000 7fe7f641b640 1 -- 172.21.15.136:0/2013884000 wait complete. 2024-04-25T11:44:46.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.248+0000 7fe7f641b640 1 Processor -- start 2024-04-25T11:44:46.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.248+0000 7fe7f641b640 1 -- start start 2024-04-25T11:44:46.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:46.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33894/0 (socket says 172.21.15.136:33894) 2024-04-25T11:44:46.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 -- 172.21.15.136:0/798356592 learned_addr learned my addr 172.21.15.136:0/798356592 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:46.237 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 0x7fe7f01445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f01476c0 0x7fe7f0149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0164f30 con 0x7fe7f0074ee0 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f00ca000 con 0x7fe7f015c300 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 0x7fe7f01445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f01650b0 con 0x7fe7f01476c0 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 -- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f01476c0 msgr2=0x7fe7f0149000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f01476c0 0x7fe7f0149000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 -- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 msgr2=0x7fe7f01445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:46.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 0x7fe7f01445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.252+0000 7fe7effff640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f01540f0 con 0x7fe7f0074ee0 2024-04-25T11:44:46.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.256+0000 7fe7effff640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800be80 tx=0x7fe7e800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:46.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.256+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8019070 con 0x7fe7f0074ee0 2024-04-25T11:44:46.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.256+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e8014440 con 0x7fe7f0074ee0 2024-04-25T11:44:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.256+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8014830 con 0x7fe7f0074ee0 2024-04-25T11:44:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.260+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 0x7fe7f01445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:46.243 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.260+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f01f21a0 con 0x7fe7f0074ee0 2024-04-25T11:44:46.245 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.260+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f0148940 con 0x7fe7f0074ee0 2024-04-25T11:44:46.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.260+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f006a350 con 0x7fe7f0074ee0 2024-04-25T11:44:46.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.264+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe7e80149d0 con 0x7fe7f0074ee0 2024-04-25T11:44:46.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.264+0000 7fe7ecff9640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 0x7fe7dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:46.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.264+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fe7e8093170 con 0x7fe7f0074ee0 2024-04-25T11:44:46.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.268+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 0x7fe7dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:46.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.268+0000 7fe7ef7fe640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 0x7fe7dc071940 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40af000 tx=0x7fe7e40a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:46.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.276+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe7e805bee0 con 0x7fe7f0074ee0 2024-04-25T11:44:46.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.824+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7f0073600 con 0x7fe7f0074ee0 2024-04-25T11:44:46.812 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.828+0000 7fe7ecff9640 1 -- 172.21.15.136:0/798356592 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+20778 (secure 0 0 0) 0x7fe7e805b7e0 con 0x7fe7f0074ee0 2024-04-25T11:44:46.812 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:46.812 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:44:19.077280+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6837/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6836/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6834/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6838/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6835/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6839/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6833/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.136:6836/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.136:6832/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6834/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6832/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.157:6839/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6834/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.136:6836/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.136:6837/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.157:6838/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6837/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.157:6835/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6835/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6833/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6837/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6835/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.157:6834/694883637":"2024-04-26T11:30:48.887575+0000","172.21.15.157:6836/694883637":"2024-04-26T11:30:48.887575+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:44:46.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 msgr2=0x7fe7dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:46.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 0x7fe7dc071940 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40af000 tx=0x7fe7e40a8000 comp rx=0 tx=0).stop 2024-04-25T11:44:46.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 msgr2=0x7fe7f0153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:46.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800be80 tx=0x7fe7e800bf70 comp rx=0 tx=0).stop 2024-04-25T11:44:46.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 shutdown_connections 2024-04-25T11:44:46.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7dc06f430 0x7fe7dc071940 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7f01476c0 0x7fe7f0149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7f015c300 0x7fe7f01445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 --2- 172.21.15.136:0/798356592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7f0074ee0 0x7fe7f0153bb0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 >> 172.21.15.136:0/798356592 conn(0x7fe7f00ca5a0 msgr2=0x7fe7f006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 shutdown_connections 2024-04-25T11:44:46.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:46.832+0000 7fe7f641b640 1 -- 172.21.15.136:0/798356592 wait complete. 2024-04-25T11:44:46.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:44:46.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:44:47.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.268+0000 7fa6d4b48640 1 Processor -- start 2024-04-25T11:44:47.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.268+0000 7fa6d4b48640 1 -- start start 2024-04-25T11:44:47.258 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.272+0000 7fa6d4b48640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa6d0155fd0 0x7fa6d00c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.272+0000 7fa6d4b48640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d0070400 0x7fa6d01668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.272+0000 7fa6d4b48640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d015cea0 con 0x7fa6d01563e0 2024-04-25T11:44:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.272+0000 7fa6d4b48640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa6d00c8c50 con 0x7fa6d0155fd0 2024-04-25T11:44:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.272+0000 7fa6d4b48640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d015cbc0 con 0x7fa6d0070400 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa6d01563e0 0x7fa6d006fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:51606/0 (socket says 172.21.15.136:51606) 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 -- 172.21.15.136:0/2496093608 learned_addr learned my addr 172.21.15.136:0/2496093608 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887757413 0 0) 0x7fa6d015cea0 con 0x7fa6d01563e0 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6b8003530 con 0x7fa6d01563e0 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1853436799 0 0) 0x7fa6b8003530 con 0x7fa6d01563e0 2024-04-25T11:44:47.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d0070400 msgr2=0x7fa6d01668b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d0070400 0x7fa6d01668b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 >> v2:172.21.15.157:3300/0 conn(0x7fa6d0155fd0 msgr2=0x7fa6d00c90e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 --2- 172.21.15.136:0/2496093608 >> v2:172.21.15.157:3300/0 conn(0x7fa6d0155fd0 0x7fa6d00c90e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d00c8600 con 0x7fa6d01563e0 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fa6c4001680 con 0x7fa6d01563e0 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 >> v1:172.21.15.157:6789/0 conn(0x7fa6d01563e0 legacy=0x7fa6d006fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b80039c0 0x7fa6b8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6b8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6b8005360 0x7fa6b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00c8c50 con 0x7fa6b8005360 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d015cbc0 con 0x7fa6b8004810 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b800c2d0 con 0x7fa6b80039c0 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6b8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:47.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6ced9d640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6b8005360 0x7fa6b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 -- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b80039c0 msgr2=0x7fa6b8003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6b80039c0 0x7fa6b8003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 -- 172.21.15.136:0/2496093608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6b8005360 msgr2=0x7fa6b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6b8005360 0x7fa6b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.276+0000 7fa6cdd9b640 1 -- 172.21.15.136:0/2496093608 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b8003530 con 0x7fa6b8004810 2024-04-25T11:44:47.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6ced9d640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6b8005360 0x7fa6b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:44:47.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6cdd9b640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6b8004c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa6d01567f0 tx=0x7fa6c401cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=705a61edba3e92a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:47.265 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4007cb0 con 0x7fa6b8004810 2024-04-25T11:44:47.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6cd59a640 1 -- 172.21.15.136:0/2496093608 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c40a9020 con 0x7fa6b8004810 2024-04-25T11:44:47.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 msgr2=0x7fa6b8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:47.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2496093608 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6b8004c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa6d01567f0 tx=0x7fa6c401cc00 comp rx=0 tx=0).stop 2024-04-25T11:44:47.266 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.280+0000 7fa6ced9d640 1 -- 172.21.15.136:0/2496093608 reap_dead start 2024-04-25T11:44:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.284+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2496093608 shutdown_connections 2024-04-25T11:44:47.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.284+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2496093608 >> 172.21.15.136:0/2496093608 conn(0x7fa6d00c9d50 msgr2=0x7fa6d00ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:47.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.292+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2496093608 shutdown_connections 2024-04-25T11:44:47.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.296+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2496093608 wait complete. 2024-04-25T11:44:47.283 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 Processor -- start 2024-04-25T11:44:47.284 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- start start 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6d00cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00c74c0 0x7fa6d01f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00caa60 con 0x7fa6d00c74c0 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00ca8e0 con 0x7fa6b8004810 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d00729c0 con 0x7fa6d00c6320 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50090/0 (socket says 172.21.15.136:50090) 2024-04-25T11:44:47.285 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 -- 172.21.15.136:0/2138667030 learned_addr learned my addr 172.21.15.136:0/2138667030 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:47.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6cdd9b640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00c74c0 0x7fa6d01f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:47.286 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 -- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 msgr2=0x7fa6d00cb050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6d00cb050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 -- 172.21.15.136:0/2138667030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00c74c0 msgr2=0x7fa6d01f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00c74c0 0x7fa6d01f1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 -- 172.21.15.136:0/2138667030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d00c6a50 con 0x7fa6d00c6320 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6ced9d640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0074560 tx=0x7fa6c800b700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8014070 con 0x7fa6d00c6320 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c800e620 con 0x7fa6d00c6320 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d00cbaa0 con 0x7fa6d00c6320 2024-04-25T11:44:47.287 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d02029c0 con 0x7fa6d00c6320 2024-04-25T11:44:47.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.300+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800ea10 con 0x7fa6d00c6320 2024-04-25T11:44:47.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.304+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b40052a0 con 0x7fa6d00c6320 2024-04-25T11:44:47.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.304+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa6c800ebb0 con 0x7fa6d00c6320 2024-04-25T11:44:47.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.304+0000 7fa6beffd640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 0x7fa6b00719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:47.289 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.304+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fa6c808fec0 con 0x7fa6d00c6320 2024-04-25T11:44:47.292 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.308+0000 7fa6ce59c640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 0x7fa6b00719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:47.293 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.308+0000 7fa6ce59c640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 0x7fa6b00719e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009690 tx=0x7fa6c0006be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:47.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.312+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa6c805bfa0 con 0x7fa6d00c6320 2024-04-25T11:44:47.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.784+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa6b4005da0 con 0x7fa6d00c6320 2024-04-25T11:44:47.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.788+0000 7fa6beffd640 1 -- 172.21.15.136:0/2138667030 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7fa6c80602a0 con 0x7fa6d00c6320 2024-04-25T11:44:47.776 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 msgr2=0x7fa6b00719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 0x7fa6b00719e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009690 tx=0x7fa6c0006be0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 msgr2=0x7fa6d014b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0074560 tx=0x7fa6c800b700 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 shutdown_connections 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa6b006f4d0 0x7fa6b00719e0 secure :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009690 tx=0x7fa6c0006be0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6d00c6320 0x7fa6d014b9e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00c74c0 0x7fa6d01f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 --2- 172.21.15.136:0/2138667030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6b8004810 0x7fa6d00cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:47.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.792+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 >> 172.21.15.136:0/2138667030 conn(0x7fa6d00c9d50 msgr2=0x7fa6d006ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:47.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.796+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 shutdown_connections 2024-04-25T11:44:47.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:47.796+0000 7fa6d4b48640 1 -- 172.21.15.136:0/2138667030 wait complete. 2024-04-25T11:44:47.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:44:48.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.236+0000 7ff700685640 1 Processor -- start 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.248+0000 7ff700685640 1 -- start start 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff700685640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 0x7ff6f815a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff700685640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff6f80c8060 0x7ff6f8070400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff700685640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6f80729c0 con 0x7ff6f80c7c50 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff700685640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f8070940 con 0x7ff6f80c8060 2024-04-25T11:44:48.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff700685640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff6f8074300 con 0x7ff6f81603a0 2024-04-25T11:44:48.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff6fec22640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff6f80c8060 0x7ff6f8070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff6fec22640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff6f80c8060 0x7ff6f8070400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33922/0 (socket says 172.21.15.136:33922) 2024-04-25T11:44:48.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff6fec22640 1 -- 172.21.15.136:0/1433805274 learned_addr learned my addr 172.21.15.136:0/1433805274 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:48.238 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.252+0000 7ff6fe421640 1 --2- 172.21.15.136:0/1433805274 >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 0x7ff6f815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.256+0000 7ff6fec22640 1 -- 172.21.15.136:0/1433805274 >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 msgr2=0x7ff6f815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:48.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.256+0000 7ff6fec22640 1 --2- 172.21.15.136:0/1433805274 >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 0x7ff6f815a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.256+0000 7ff6fec22640 1 -- 172.21.15.136:0/1433805274 >> v1:172.21.15.157:6789/0 conn(0x7ff6f80c7c50 legacy=0x7ff6f806fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.256+0000 7ff6fd41f640 1 -- 172.21.15.136:0/1433805274 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399436284 0 0) 0x7ff6f80729c0 con 0x7ff6f80c7c50 2024-04-25T11:44:48.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.256+0000 7ff6fec22640 1 -- 172.21.15.136:0/1433805274 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f8072c80 con 0x7ff6f80c8060 2024-04-25T11:44:48.242 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.260+0000 7ff6fe421640 1 --2- 172.21.15.136:0/1433805274 >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 0x7ff6f815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.260+0000 7ff6fec22640 1 --2- 172.21.15.136:0/1433805274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f80c8060 0x7ff6f8070400 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8074f20 tx=0x7ff6e801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cc49ba0ac9307b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.260+0000 7ff6fd41f640 1 -- 172.21.15.136:0/1433805274 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80b5070 con 0x7ff6f80c8060 2024-04-25T11:44:48.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.260+0000 7ff6fd41f640 1 -- 172.21.15.136:0/1433805274 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e80a8a10 con 0x7ff6f80c8060 2024-04-25T11:44:48.246 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.260+0000 7ff6fd41f640 1 -- 172.21.15.136:0/1433805274 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80a8de0 con 0x7ff6f80c8060 2024-04-25T11:44:48.247 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 -- 172.21.15.136:0/1433805274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f80c8060 msgr2=0x7ff6f8070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.248 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 --2- 172.21.15.136:0/1433805274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f80c8060 0x7ff6f8070400 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8074f20 tx=0x7ff6e801ed10 comp rx=0 tx=0).stop 2024-04-25T11:44:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 -- 172.21.15.136:0/1433805274 shutdown_connections 2024-04-25T11:44:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 --2- 172.21.15.136:0/1433805274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f80c8060 0x7ff6f8070400 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 --2- 172.21.15.136:0/1433805274 >> v2:172.21.15.157:3300/0 conn(0x7ff6f81603a0 0x7ff6f815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.264+0000 7ff700685640 1 -- 172.21.15.136:0/1433805274 >> 172.21.15.136:0/1433805274 conn(0x7ff6f80c9d50 msgr2=0x7ff6f80ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.268+0000 7ff700685640 1 -- 172.21.15.136:0/1433805274 shutdown_connections 2024-04-25T11:44:48.253 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.268+0000 7ff700685640 1 -- 172.21.15.136:0/1433805274 wait complete. 2024-04-25T11:44:48.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.272+0000 7ff700685640 1 Processor -- start 2024-04-25T11:44:48.256 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.272+0000 7ff700685640 1 -- start start 2024-04-25T11:44:48.260 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.272+0000 7ff700685640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.272+0000 7ff700685640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6f80c8060 0x7ff6f815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 0x7ff6f80c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37372/0 (socket says 172.21.15.136:37372) 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 -- 172.21.15.136:0/1056725097 learned_addr learned my addr 172.21.15.136:0/1056725097 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fdc20640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6f80c8060 0x7ff6f815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8073520 con 0x7ff6f81603a0 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8073240 con 0x7ff6f80c7c50 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f8073800 con 0x7ff6f80c8060 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 -- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6f80c8060 msgr2=0x7ff6f815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6f80c8060 0x7ff6f815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 -- 172.21.15.136:0/1056725097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 msgr2=0x7ff6f80c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fec22640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 0x7ff6f80c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 0x7ff6f80c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f8164170 con 0x7ff6f80c7c50 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fec22640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 0x7ff6f80c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6fe421640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c820 tx=0x7ff6f400cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4015070 con 0x7ff6f80c7c50 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f4007a30 con 0x7ff6f80c7c50 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4007e40 con 0x7ff6f80c7c50 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f8163470 con 0x7ff6f80c7c50 2024-04-25T11:44:48.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.276+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f8162310 con 0x7ff6f80c7c50 2024-04-25T11:44:48.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.280+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff6f4018070 con 0x7ff6f80c7c50 2024-04-25T11:44:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.280+0000 7ff6eeffd640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 0x7ff6dc093950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.280+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7ff6f408fa30 con 0x7ff6f80c7c50 2024-04-25T11:44:48.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.284+0000 7ff6fdc20640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 0x7ff6dc093950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:48.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.288+0000 7ff6fdc20640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 0x7ff6dc093950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff6f815be10 tx=0x7ff6f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:48.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.292+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e00052a0 con 0x7ff6f80c7c50 2024-04-25T11:44:48.288 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.304+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff6f405ba10 con 0x7ff6f80c7c50 2024-04-25T11:44:48.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.744+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff6e0005da0 con 0x7ff6f80c7c50 2024-04-25T11:44:48.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.748+0000 7ff6eeffd640 1 -- 172.21.15.136:0/1056725097 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) ==== 104+0+11 (secure 0 0 0) 0x7ff6f405fd10 con 0x7ff6f80c7c50 2024-04-25T11:44:48.743 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:44:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.776+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 msgr2=0x7ff6dc093950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.776+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 0x7ff6dc093950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff6f815be10 tx=0x7ff6f0009040 comp rx=0 tx=0).stop 2024-04-25T11:44:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.776+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 msgr2=0x7ff6f8163c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.776+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c820 tx=0x7ff6f400cce0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 shutdown_connections 2024-04-25T11:44:48.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6f81603a0 0x7ff6f80c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff6dc091440 0x7ff6dc093950 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff6f80c8060 0x7ff6f815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 --2- 172.21.15.136:0/1056725097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6f80c7c50 0x7ff6f8163c30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:48.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 >> 172.21.15.136:0/1056725097 conn(0x7ff6f80c9d50 msgr2=0x7ff6f815ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:48.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 shutdown_connections 2024-04-25T11:44:48.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:48.780+0000 7ff700685640 1 -- 172.21.15.136:0/1056725097 wait complete. 2024-04-25T11:44:48.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:44:49.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.156+0000 7f5f5e3d5640 1 Processor -- start 2024-04-25T11:44:49.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.156+0000 7f5f5e3d5640 1 -- start start 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5e3d5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5e3d5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 0x7f5f58073ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5e3d5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f58161140 con 0x7f5f5815a110 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50096/0 (socket says 172.21.15.136:50096) 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 -- 172.21.15.136:0/116768000 learned_addr learned my addr 172.21.15.136:0/116768000 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5f58072f50 con 0x7f5f580736c0 2024-04-25T11:44:49.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5f58073230 con 0x7f5f580688b0 2024-04-25T11:44:49.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 -- 172.21.15.136:0/116768000 >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 msgr2=0x7f5f58073ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:49.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5cbd2640 1 --2- 172.21.15.136:0/116768000 >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 0x7f5f58073ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.146 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 --2- 172.21.15.136:0/116768000 >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 0x7f5f58073ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 -- 172.21.15.136:0/116768000 >> v1:172.21.15.157:6789/0 conn(0x7f5f5815a110 legacy=0x7f5f5815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5d3d3640 1 -- 172.21.15.136:0/116768000 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5806f1e0 con 0x7f5f580688b0 2024-04-25T11:44:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.160+0000 7f5f5cbd2640 1 --2- 172.21.15.136:0/116768000 >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 0x7f5f58073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.168+0000 7f5f5d3d3640 1 --2- 172.21.15.136:0/116768000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5f580730d0 tx=0x7f5f5401c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb77dbd506315ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.168+0000 7f5f4ffff640 1 -- 172.21.15.136:0/116768000 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f540b1070 con 0x7f5f580688b0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.168+0000 7f5f4ffff640 1 -- 172.21.15.136:0/116768000 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f540ac620 con 0x7f5f580688b0 2024-04-25T11:44:49.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.168+0000 7f5f4ffff640 1 -- 172.21.15.136:0/116768000 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f540ac9f0 con 0x7f5f580688b0 2024-04-25T11:44:49.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 msgr2=0x7f5f5806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/116768000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5f580730d0 tx=0x7f5f5401c460 comp rx=0 tx=0).stop 2024-04-25T11:44:49.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 shutdown_connections 2024-04-25T11:44:49.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/116768000 >> v2:172.21.15.157:3300/0 conn(0x7f5f580736c0 0x7f5f58073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/116768000 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580688b0 0x7f5f5806d850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.176+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 >> 172.21.15.136:0/116768000 conn(0x7f5f5806e6c0 msgr2=0x7f5f5806ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.180+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 shutdown_connections 2024-04-25T11:44:49.163 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.180+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/116768000 wait complete. 2024-04-25T11:44:49.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.180+0000 7f5f5e3d5640 1 Processor -- start 2024-04-25T11:44:49.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 -- start start 2024-04-25T11:44:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580736c0 0x7f5f581e6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 0x7f5f5815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5f58160d70 con 0x7f5f580688b0 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5d3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5d3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33944/0 (socket says 172.21.15.136:33944) 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5d3d3640 1 -- 172.21.15.136:0/3186958150 learned_addr learned my addr 172.21.15.136:0/3186958150 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 0x7f5f5815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5806f7a0 con 0x7f5f5815a110 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5f58160ef0 con 0x7f5f580736c0 2024-04-25T11:44:49.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 -- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580736c0 msgr2=0x7f5f581e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580736c0 0x7f5f581e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 -- 172.21.15.136:0/3186958150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 msgr2=0x7f5f581f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5dbd4640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f581fa0e0 con 0x7f5f5815a110 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.184+0000 7f5f5d3d3640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:49.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.188+0000 7f5f5dbd4640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 0x7f5f5815ca50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c060 tx=0x7f5f5000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.188+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50018030 con 0x7f5f5815a110 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.188+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f58153950 con 0x7f5f5815a110 2024-04-25T11:44:49.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.188+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f581fa380 con 0x7f5f5815a110 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f50003980 con 0x7f5f5815a110 2024-04-25T11:44:49.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50003d50 con 0x7f5f5815a110 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5f5001b070 con 0x7f5f5815a110 2024-04-25T11:44:49.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f4dffb640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 0x7f5f3c071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f5f500920d0 con 0x7f5f5815a110 2024-04-25T11:44:49.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.192+0000 7f5f5d3d3640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 0x7f5f3c071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:49.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.196+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f380052a0 con 0x7f5f5815a110 2024-04-25T11:44:49.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.196+0000 7f5f5d3d3640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 0x7f5f3c071960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c10 tx=0x7f5f54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:49.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.204+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5f5005e780 con 0x7f5f5815a110 2024-04-25T11:44:49.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.680+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5f38005da0 con 0x7f5f5815a110 2024-04-25T11:44:49.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.680+0000 7f5f4dffb640 1 -- 172.21.15.136:0/3186958150 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7f5f5005e080 con 0x7f5f5815a110 2024-04-25T11:44:49.667 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:44:49.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 msgr2=0x7f5f3c071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 0x7f5f3c071960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c10 tx=0x7f5f54008040 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 msgr2=0x7f5f5815ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 0x7f5f5815ca50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000c060 tx=0x7f5f5000c520 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 shutdown_connections 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5f5815a110 0x7f5f5815ca50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f5f3c06f450 0x7f5f3c071960 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5f580736c0 0x7f5f581e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 --2- 172.21.15.136:0/3186958150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f580688b0 0x7f5f581f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.684+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 >> 172.21.15.136:0/3186958150 conn(0x7f5f5806e6c0 msgr2=0x7f5f581579c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.688+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 shutdown_connections 2024-04-25T11:44:49.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:49.688+0000 7f5f5e3d5640 1 -- 172.21.15.136:0/3186958150 wait complete. 2024-04-25T11:44:49.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:44:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.008+0000 7f2d4a453640 1 Processor -- start 2024-04-25T11:44:49.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.008+0000 7f2d4a453640 1 -- start start 2024-04-25T11:44:50.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d3c0e2890 0x7f2d3c0e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2d3c0df2f0 0x7f2d3c0df700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50102/0 (socket says 172.21.15.136:50102) 2024-04-25T11:44:50.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 -- 172.21.15.136:0/2806923584 learned_addr learned my addr 172.21.15.136:0/2806923584 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:50.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3c0e3510 con 0x7f2d3c0df2f0 2024-04-25T11:44:50.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3c0e37f0 con 0x7f2d3c0e2890 2024-04-25T11:44:50.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3c04b730 con 0x7f2d3c0060a0 2024-04-25T11:44:50.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 -- 172.21.15.136:0/2806923584 >> v2:172.21.15.157:3300/0 conn(0x7f2d3c0e2890 msgr2=0x7f2d3c0e2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:50.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 --2- 172.21.15.136:0/2806923584 >> v2:172.21.15.157:3300/0 conn(0x7f2d3c0e2890 0x7f2d3c0e2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 -- 172.21.15.136:0/2806923584 >> v2:172.21.15.136:3300/0 conn(0x7f2d3c0df2f0 msgr2=0x7f2d3c0df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:50.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 --2- 172.21.15.136:0/2806923584 >> v2:172.21.15.136:3300/0 conn(0x7f2d3c0df2f0 0x7f2d3c0df700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.016+0000 7f2d43fff640 1 -- 172.21.15.136:0/2806923584 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3c04ba10 con 0x7f2d3c0060a0 2024-04-25T11:44:50.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.020+0000 7f2d43fff640 1 --2- 172.21.15.136:0/2806923584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c04cda0 tx=0x7f2d3801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e632fc2921c99d1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.020+0000 7f2d42ffd640 1 -- 172.21.15.136:0/2806923584 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380b0070 con 0x7f2d3c0060a0 2024-04-25T11:44:50.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.020+0000 7f2d42ffd640 1 -- 172.21.15.136:0/2806923584 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d380ab600 con 0x7f2d3c0060a0 2024-04-25T11:44:50.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d42ffd640 1 -- 172.21.15.136:0/2806923584 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380ab9b0 con 0x7f2d3c0060a0 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 msgr2=0x7f2d3c0064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 --2- 172.21.15.136:0/2806923584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c04cda0 tx=0x7f2d3801c560 comp rx=0 tx=0).stop 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 shutdown_connections 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 --2- 172.21.15.136:0/2806923584 >> v2:172.21.15.136:3300/0 conn(0x7f2d3c0df2f0 0x7f2d3c0df700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 --2- 172.21.15.136:0/2806923584 >> v2:172.21.15.157:3300/0 conn(0x7f2d3c0e2890 0x7f2d3c0e2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 --2- 172.21.15.136:0/2806923584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0064b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 >> 172.21.15.136:0/2806923584 conn(0x7f2d3c0503c0 msgr2=0x7f2d3c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 shutdown_connections 2024-04-25T11:44:50.009 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- 172.21.15.136:0/2806923584 wait complete. 2024-04-25T11:44:50.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 Processor -- start 2024-04-25T11:44:50.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.024+0000 7f2d4a453640 1 -- start start 2024-04-25T11:44:50.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d3c0df2f0 0x7f2d3c0cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50114/0 (socket says 172.21.15.136:50114) 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 -- 172.21.15.136:0/1045143580 learned_addr learned my addr 172.21.15.136:0/1045143580 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d437fe640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d3c0df2f0 0x7f2d3c0cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d3c0e2890 0x7f2d3c0cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c0e01d0 con 0x7f2d3c0e2890 2024-04-25T11:44:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c0e0050 con 0x7f2d3c0df2f0 2024-04-25T11:44:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d489f0640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d3c0e2890 0x7f2d3c0cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3c0473f0 con 0x7f2d3c0060a0 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 -- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d3c0df2f0 msgr2=0x7f2d3c0cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d3c0df2f0 0x7f2d3c0cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 -- 172.21.15.136:0/1045143580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d3c0e2890 msgr2=0x7f2d3c0cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d3c0e2890 0x7f2d3c0cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.028+0000 7f2d43fff640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3c17cc90 con 0x7f2d3c0060a0 2024-04-25T11:44:50.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d43fff640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2d380ae040 tx=0x7f2d3801ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380b0040 con 0x7f2d3c0060a0 2024-04-25T11:44:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d380bc070 con 0x7f2d3c0060a0 2024-04-25T11:44:50.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380b6840 con 0x7f2d3c0060a0 2024-04-25T11:44:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3c17cf30 con 0x7f2d3c0060a0 2024-04-25T11:44:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3c050160 con 0x7f2d3c0060a0 2024-04-25T11:44:50.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.032+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d3c005580 con 0x7f2d3c0060a0 2024-04-25T11:44:50.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.036+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2d380b69e0 con 0x7f2d3c0060a0 2024-04-25T11:44:50.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.036+0000 7f2d40ff9640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 0x7f2d30071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.036+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f2d3812abb0 con 0x7f2d3c0060a0 2024-04-25T11:44:50.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.036+0000 7f2d437fe640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 0x7f2d30071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.022 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.036+0000 7f2d437fe640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 0x7f2d30071960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000b70 tx=0x7f2d4406f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.048+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2d38016150 con 0x7f2d3c0060a0 2024-04-25T11:44:50.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.412+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2d3c031bb0 con 0x7f2d3c0060a0 2024-04-25T11:44:50.399 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.416+0000 7f2d40ff9640 1 -- 172.21.15.136:0/1045143580 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v40) ==== 103+0+11 (secure 0 0 0) 0x7f2d380ff360 con 0x7f2d3c0060a0 2024-04-25T11:44:50.404 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:44:50.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 msgr2=0x7f2d30071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 0x7f2d30071960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000b70 tx=0x7f2d4406f040 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 msgr2=0x7f2d3c0c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2d380ae040 tx=0x7f2d3801ada0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 shutdown_connections 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d3c0e2890 0x7f2d3c0cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d3c0df2f0 0x7f2d3c0cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f2d3006f450 0x7f2d30071960 secure :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000b70 tx=0x7f2d4406f040 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 --2- 172.21.15.136:0/1045143580 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d3c0060a0 0x7f2d3c0c69a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.432+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 >> 172.21.15.136:0/1045143580 conn(0x7f2d3c0503c0 msgr2=0x7f2d3c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.436+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 shutdown_connections 2024-04-25T11:44:50.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.436+0000 7f2d4a453640 1 -- 172.21.15.136:0/1045143580 wait complete. 2024-04-25T11:44:50.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T11:44:50.745 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.760+0000 7fbb8e162640 1 Processor -- start 2024-04-25T11:44:50.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.760+0000 7fbb8e162640 1 -- start start 2024-04-25T11:44:50.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8e162640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbb80045bf0 0x7fbb80046000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8e162640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb80007c50 con 0x7fbb800472e0 2024-04-25T11:44:50.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8e162640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb80007100 con 0x7fbb800ec040 2024-04-25T11:44:50.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8e162640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fbb800e3ab0 con 0x7fbb80045bf0 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8c95f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbb80045bf0 0x7fbb80046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8d160640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbb800472e0 0x7fbb800e3f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:39616/0 (socket says 172.21.15.136:39616) 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8d160640 1 -- 172.21.15.136:0/2836511721 learned_addr learned my addr 172.21.15.136:0/2836511721 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8c95f640 1 -- 172.21.15.136:0/2836511721 >> v1:172.21.15.157:6789/0 conn(0x7fbb800ec040 legacy=0x7fbb800ee460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8c95f640 1 -- 172.21.15.136:0/2836511721 >> v1:172.21.15.136:6789/0 conn(0x7fbb800472e0 legacy=0x7fbb800e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.764+0000 7fbb8c95f640 1 -- 172.21.15.136:0/2836511721 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb80007f10 con 0x7fbb80045bf0 2024-04-25T11:44:50.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8c95f640 1 --2- 172.21.15.136:0/2836511721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb80045bf0 0x7fbb80046000 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097c0 tx=0x7fbb8401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1584d693b2341a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb7ffff640 1 -- 172.21.15.136:0/2836511721 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840b0070 con 0x7fbb80045bf0 2024-04-25T11:44:50.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb7ffff640 1 -- 172.21.15.136:0/2836511721 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb840ab600 con 0x7fbb80045bf0 2024-04-25T11:44:50.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb7ffff640 1 -- 172.21.15.136:0/2836511721 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840ab9d0 con 0x7fbb80045bf0 2024-04-25T11:44:50.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 -- 172.21.15.136:0/2836511721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb80045bf0 msgr2=0x7fbb80046000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 --2- 172.21.15.136:0/2836511721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb80045bf0 0x7fbb80046000 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbb840097c0 tx=0x7fbb8401c400 comp rx=0 tx=0).stop 2024-04-25T11:44:50.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 -- 172.21.15.136:0/2836511721 shutdown_connections 2024-04-25T11:44:50.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 --2- 172.21.15.136:0/2836511721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb80045bf0 0x7fbb80046000 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 -- 172.21.15.136:0/2836511721 >> 172.21.15.136:0/2836511721 conn(0x7fbb80050c50 msgr2=0x7fbb80051060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:50.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.768+0000 7fbb8e162640 1 -- 172.21.15.136:0/2836511721 shutdown_connections 2024-04-25T11:44:50.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.772+0000 7fbb8e162640 1 -- 172.21.15.136:0/2836511721 wait complete. 2024-04-25T11:44:50.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.772+0000 7fbb8e162640 1 Processor -- start 2024-04-25T11:44:50.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.772+0000 7fbb8e162640 1 -- start start 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb800ec040 0x7fbb800d64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 0x7fbb80186690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbb800e36d0 con 0x7fbb800e78d0 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb80036690 con 0x7fbb800472e0 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8e162640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb80036ca0 con 0x7fbb800ec040 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d160640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d160640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37418/0 (socket says 172.21.15.136:37418) 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d160640 1 -- 172.21.15.136:0/3841696486 learned_addr learned my addr 172.21.15.136:0/3841696486 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 0x7fbb80186690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8c95f640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb800ec040 0x7fbb800d64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 -- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb800ec040 msgr2=0x7fbb800d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb800ec040 0x7fbb800d64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 -- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 msgr2=0x7fbb800e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 -- 172.21.15.136:0/3841696486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb800e2490 con 0x7fbb800e78d0 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d160640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:50.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.776+0000 7fbb8d961640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 0x7fbb80186690 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b3e0 tx=0x7fbb7800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.780+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb78015020 con 0x7fbb800e78d0 2024-04-25T11:44:50.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.780+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb78011830 con 0x7fbb800e78d0 2024-04-25T11:44:50.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.780+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb78011c20 con 0x7fbb800e78d0 2024-04-25T11:44:50.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.780+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb800cc110 con 0x7fbb800e78d0 2024-04-25T11:44:50.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.780+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb800cc6d0 con 0x7fbb800e78d0 2024-04-25T11:44:50.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.784+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb800e2660 con 0x7fbb800e78d0 2024-04-25T11:44:50.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.788+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbb78018020 con 0x7fbb800e78d0 2024-04-25T11:44:50.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.788+0000 7fbb7dffb640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 0x7fbb6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:50.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.792+0000 7fbb8d160640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 0x7fbb6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.792+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fbb78011dc0 con 0x7fbb800e78d0 2024-04-25T11:44:50.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.792+0000 7fbb8d160640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 0x7fbb6c071910 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbb8807cdc0 tx=0x7fbb8808f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:50.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:50.792+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb7801f020 con 0x7fbb800e78d0 2024-04-25T11:44:51.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.340+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbb800059d0 con 0x7fbb800e78d0 2024-04-25T11:44:51.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.340+0000 7fbb7dffb640 1 -- 172.21.15.136:0/3841696486 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 28 entries v40) ==== 81+0+1728 (secure 0 0 0) 0x7fbb7805b7c0 con 0x7fbb800e78d0 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/1501623433 2024-04-26T11:41:59.046316+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/1501623433 2024-04-26T11:41:59.046316+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/3576995854 2024-04-26T11:36:58.974013+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/3576995854 2024-04-26T11:36:58.974013+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/3688751078 2024-04-26T11:33:28.923953+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6836/3463942074 2024-04-26T11:38:18.992149+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6834/430807169 2024-04-26T11:23:23.787444+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6838/3020784477 2024-04-26T11:42:54.061351+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6835/430807169 2024-04-26T11:23:23.787444+0000 2024-04-25T11:44:51.325 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6839/3020784477 2024-04-26T11:42:54.061351+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6833/3353376405 2024-04-26T11:25:58.820255+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6836/1740633057 2024-04-26T11:44:19.077214+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6832/906354453 2024-04-26T11:39:24.008268+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6834/3463942074 2024-04-26T11:38:18.992149+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6832/3353376405 2024-04-26T11:25:58.820255+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6839/1671830331 2024-04-26T11:32:38.912872+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6834/3406304193 2024-04-26T11:35:58.955925+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6836/1368493675 2024-04-26T11:28:33.853864+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6837/1740633057 2024-04-26T11:44:19.077214+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6838/1671830331 2024-04-26T11:32:38.912872+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6837/1368493675 2024-04-26T11:28:33.853864+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/3688751078 2024-04-26T11:33:28.923953+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/281980057 2024-04-26T11:30:08.877388+0000 2024-04-25T11:44:51.326 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6833/906354453 2024-04-26T11:39:24.008268+0000 2024-04-25T11:44:51.327 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/281980057 2024-04-26T11:30:08.877388+0000 2024-04-25T11:44:51.327 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6835/3406304193 2024-04-26T11:35:58.955925+0000 2024-04-25T11:44:51.327 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6834/694883637 2024-04-26T11:30:48.887575+0000 2024-04-25T11:44:51.327 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6836/694883637 2024-04-26T11:30:48.887575+0000 2024-04-25T11:44:51.327 INFO:teuthology.orchestra.run.smithi136.stderr:listed 28 entries 2024-04-25T11:44:51.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.348+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 msgr2=0x7fbb6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.348+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 0x7fbb6c071910 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbb8807cdc0 tx=0x7fbb8808f040 comp rx=0 tx=0).stop 2024-04-25T11:44:51.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.348+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 msgr2=0x7fbb80186690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.348+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 0x7fbb80186690 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b3e0 tx=0x7fbb7800b8a0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 shutdown_connections 2024-04-25T11:44:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb800e78d0 0x7fbb80186690 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbb6c06f400 0x7fbb6c071910 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb800ec040 0x7fbb800d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 --2- 172.21.15.136:0/3841696486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800472e0 0x7fbb800e1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 >> 172.21.15.136:0/3841696486 conn(0x7fbb80050c50 msgr2=0x7fbb800d8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 shutdown_connections 2024-04-25T11:44:51.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.352+0000 7fbb8e162640 1 -- 172.21.15.136:0/3841696486 wait complete. 2024-04-25T11:44:51.392 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-25T11:44:51.392 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T11:44:51.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:44:51.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.820+0000 7f75fa0be640 1 Processor -- start 2024-04-25T11:44:51.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.820+0000 7f75fa0be640 1 -- start start 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75f40c4c80 0x7f75f40c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 0x7f75f40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f75f41597c0 con 0x7f75f40c4c80 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f75f4152ef0 con 0x7f75f40c6ca0 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75fa0be640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75f4158bb0 con 0x7f75f4152ae0 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f3fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 0x7f75f40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f3fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 0x7f75f40c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37422/0 (socket says 172.21.15.136:37422) 2024-04-25T11:44:51.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f3fff640 1 -- 172.21.15.136:0/2521310125 learned_addr learned my addr 172.21.15.136:0/2521310125 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2521310125 >> v2:172.21.15.136:3300/0 conn(0x7f75f40c4c80 0x7f75f40c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 -- 172.21.15.136:0/2521310125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 msgr2=0x7f75f406d910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2521310125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f406d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 -- 172.21.15.136:0/2521310125 >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 msgr2=0x7f75f40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2521310125 >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 0x7f75f40c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.824+0000 7f75f2ffd640 1 -- 172.21.15.136:0/2521310125 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f4158e90 con 0x7f75f40c4c80 2024-04-25T11:44:51.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.828+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2521310125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c4c80 0x7f75f40c5090 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f75f406ff50 tx=0x7f75e401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3ae54d005f6f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.828+0000 7f75f27fc640 1 -- 172.21.15.136:0/2521310125 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40b0070 con 0x7f75f40c4c80 2024-04-25T11:44:51.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.828+0000 7f75f27fc640 1 -- 172.21.15.136:0/2521310125 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e40ab600 con 0x7f75f40c4c80 2024-04-25T11:44:51.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.828+0000 7f75f27fc640 1 -- 172.21.15.136:0/2521310125 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40ab9d0 con 0x7f75f40c4c80 2024-04-25T11:44:51.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.832+0000 7f75fa0be640 1 -- 172.21.15.136:0/2521310125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c4c80 msgr2=0x7f75f40c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.815 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.832+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2521310125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c4c80 0x7f75f40c5090 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f75f406ff50 tx=0x7f75e401c4a0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 -- 172.21.15.136:0/2521310125 shutdown_connections 2024-04-25T11:44:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2521310125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c4c80 0x7f75f40c5090 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2521310125 >> v2:172.21.15.157:3300/0 conn(0x7f75f40c6ca0 0x7f75f40c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2521310125 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 -- 172.21.15.136:0/2521310125 >> 172.21.15.136:0/2521310125 conn(0x7f75f40ca280 msgr2=0x7f75f40ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:51.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 -- 172.21.15.136:0/2521310125 shutdown_connections 2024-04-25T11:44:51.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.836+0000 7f75fa0be640 1 -- 172.21.15.136:0/2521310125 wait complete. 2024-04-25T11:44:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.840+0000 7f75fa0be640 1 Processor -- start 2024-04-25T11:44:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.840+0000 7f75fa0be640 1 -- start start 2024-04-25T11:44:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 0x7f75f413cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37424/0 (socket says 172.21.15.136:37424) 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 0x7f75f413cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 0x7f75f413cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:33984/0 (socket says 172.21.15.136:33984) 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 -- 172.21.15.136:0/2190743998 learned_addr learned my addr 172.21.15.136:0/2190743998 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f413d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4073ae0 con 0x7f75f40c6ca0 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4073960 con 0x7f75f40c4c80 2024-04-25T11:44:51.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f75f415c810 con 0x7f75f4152ae0 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f3fff640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f413d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 -- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 msgr2=0x7f75f413d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f413d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 -- 172.21.15.136:0/2190743998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 msgr2=0x7f75f413cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 0x7f75f413cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.844+0000 7f75f37fe640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40c4090 con 0x7f75f40c4c80 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.848+0000 7f75f37fe640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b410 tx=0x7f75e800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.848+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e801d070 con 0x7f75f40c4c80 2024-04-25T11:44:51.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.848+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e8010a30 con 0x7f75f40c4c80 2024-04-25T11:44:51.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.848+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8010de0 con 0x7f75f40c4c80 2024-04-25T11:44:51.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.848+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f41f4d20 con 0x7f75f40c4c80 2024-04-25T11:44:51.835 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.852+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f415b910 con 0x7f75f40c4c80 2024-04-25T11:44:51.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.852+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f406a2e0 con 0x7f75f40c4c80 2024-04-25T11:44:51.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.852+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f75e8020070 con 0x7f75f40c4c80 2024-04-25T11:44:51.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.864+0000 7f75dbfff640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 0x7f75e0071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:51.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.864+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f75e8093690 con 0x7f75f40c4c80 2024-04-25T11:44:51.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.864+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f75e805b970 con 0x7f75f40c4c80 2024-04-25T11:44:51.848 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.864+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 0x7f75e0071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:51.849 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:51.864+0000 7f75f2ffd640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 0x7f75e0071990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75e40ad000 tx=0x7f75e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:52.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.364+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75f40c7f10 con 0x7f75f40c4c80 2024-04-25T11:44:52.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.368+0000 7f75dbfff640 1 -- 172.21.15.136:0/2190743998 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+20778 (secure 0 0 0) 0x7f75e805f6e0 con 0x7f75f40c4c80 2024-04-25T11:44:52.354 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:44:52.355 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:44:19.077280+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6837/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6836/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6834/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6838/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6835/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6839/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6833/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.136:6836/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.136:6832/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6834/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6832/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.157:6839/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6834/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.136:6836/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.136:6837/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.157:6838/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6837/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.157:6835/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6835/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6833/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6837/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6835/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.157:6834/694883637":"2024-04-26T11:30:48.887575+0000","172.21.15.157:6836/694883637":"2024-04-26T11:30:48.887575+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:44:52.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.376+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 msgr2=0x7f75e0071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:52.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.376+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 0x7f75e0071990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75e40ad000 tx=0x7f75e4008040 comp rx=0 tx=0).stop 2024-04-25T11:44:52.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.376+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 msgr2=0x7f75f41f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:52.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.376+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b410 tx=0x7f75e800b8d0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 shutdown_connections 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f75e006f480 0x7f75e0071990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f75f4152ae0 0x7f75f413d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f40c6ca0 0x7f75f413cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 --2- 172.21.15.136:0/2190743998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f75f40c4c80 0x7f75f41f3fe0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.380+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 >> 172.21.15.136:0/2190743998 conn(0x7f75f40ca280 msgr2=0x7f75f4148310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:52.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.384+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 shutdown_connections 2024-04-25T11:44:52.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.384+0000 7f75fa0be640 1 -- 172.21.15.136:0/2190743998 wait complete. 2024-04-25T11:44:52.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T11:20:06.800284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T11:20:17.084245+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T11:20:18.360664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T11:20:22.053112+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T11:44:52.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T11:44:52.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.860+0000 7fda1cf53640 1 Processor -- start 2024-04-25T11:44:52.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.860+0000 7fda1cf53640 1 -- start start 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1cf53640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fda180c2e00 0x7fda18068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1cf53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1cf53640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda180c3210 con 0x7fda1815bdc0 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1cf53640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fda1815c5e0 con 0x7fda180c2e00 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1cf53640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fda1815cae0 con 0x7fda1815c1d0 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50158/0 (socket says 172.21.15.136:50158) 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 -- 172.21.15.136:0/3810167287 learned_addr learned my addr 172.21.15.136:0/3810167287 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda1659c640 1 --2- 172.21.15.136:0/3810167287 >> v2:172.21.15.157:3300/0 conn(0x7fda180c2e00 0x7fda18068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:52.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 -- 172.21.15.136:0/3810167287 >> v2:172.21.15.157:3300/0 conn(0x7fda180c2e00 msgr2=0x7fda18068590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:52.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 --2- 172.21.15.136:0/3810167287 >> v2:172.21.15.157:3300/0 conn(0x7fda180c2e00 0x7fda18068590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 -- 172.21.15.136:0/3810167287 >> v1:172.21.15.157:6789/0 conn(0x7fda1815bdc0 legacy=0x7fda180749a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:52.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.864+0000 7fda16d9d640 1 -- 172.21.15.136:0/3810167287 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1815c800 con 0x7fda1815c1d0 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda16d9d640 1 --2- 172.21.15.136:0/3810167287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fda180cb440 tx=0x7fda1001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40be4bf917e3a9fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1559a640 1 -- 172.21.15.136:0/3810167287 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100a5550 con 0x7fda1815c1d0 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1559a640 1 -- 172.21.15.136:0/3810167287 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda100a56f0 con 0x7fda1815c1d0 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1559a640 1 -- 172.21.15.136:0/3810167287 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100a5aa0 con 0x7fda1815c1d0 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 -- 172.21.15.136:0/3810167287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 msgr2=0x7fda180c4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:52.858 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 --2- 172.21.15.136:0/3810167287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fda180cb440 tx=0x7fda1001c8d0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 -- 172.21.15.136:0/3810167287 shutdown_connections 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 --2- 172.21.15.136:0/3810167287 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda1815c1d0 0x7fda180c4340 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 --2- 172.21.15.136:0/3810167287 >> v2:172.21.15.157:3300/0 conn(0x7fda180c2e00 0x7fda18068590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 -- 172.21.15.136:0/3810167287 >> 172.21.15.136:0/3810167287 conn(0x7fda180ca5a0 msgr2=0x7fda180ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 -- 172.21.15.136:0/3810167287 shutdown_connections 2024-04-25T11:44:52.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.872+0000 7fda1cf53640 1 -- 172.21.15.136:0/3810167287 wait complete. 2024-04-25T11:44:52.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.880+0000 7fda1cf53640 1 Processor -- start 2024-04-25T11:44:52.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.880+0000 7fda1cf53640 1 -- start start 2024-04-25T11:44:52.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda1815bdc0 0x7fda181e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50174/0 (socket says 172.21.15.136:50174) 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 -- 172.21.15.136:0/475395724 learned_addr learned my addr 172.21.15.136:0/475395724 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda1815c1d0 0x7fda18147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fda180ca000 con 0x7fda1815bdc0 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda16d9d640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda1815c1d0 0x7fda18147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda180c9d20 con 0x7fda1815c1d0 2024-04-25T11:44:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fda180750b0 con 0x7fda180c2e00 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 -- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda1815c1d0 msgr2=0x7fda18147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda1815c1d0 0x7fda18147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 -- 172.21.15.136:0/475395724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda1815bdc0 msgr2=0x7fda181e4b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda1815bdc0 0x7fda181e4b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.884+0000 7fda1659c640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda181f4d20 con 0x7fda180c2e00 2024-04-25T11:44:52.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda1659c640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fda0801f680 tx=0x7fda0801fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:52.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08028070 con 0x7fda180c2e00 2024-04-25T11:44:52.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda0801d940 con 0x7fda180c2e00 2024-04-25T11:44:52.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda180cd0b0 con 0x7fda180c2e00 2024-04-25T11:44:52.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda18154770 con 0x7fda180c2e00 2024-04-25T11:44:52.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0801dcf0 con 0x7fda180c2e00 2024-04-25T11:44:52.874 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.888+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9fc0052a0 con 0x7fda180c2e00 2024-04-25T11:44:52.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.892+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda0802b070 con 0x7fda180c2e00 2024-04-25T11:44:52.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.892+0000 7fda06ffd640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 0x7fd9f80719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:52.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.892+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fda080a11a0 con 0x7fda180c2e00 2024-04-25T11:44:52.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.896+0000 7fda15d9b640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 0x7fd9f80719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.896+0000 7fda15d9b640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 0x7fd9f80719e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fda181553b0 tx=0x7fda0c006b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:52.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:52.900+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fda0806d1f0 con 0x7fda180c2e00 2024-04-25T11:44:53.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.316+0000 7fda1cf53640 1 -- 172.21.15.136:0/475395724 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd9fc005860 con 0x7fda180c2e00 2024-04-25T11:44:53.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.320+0000 7fda06ffd640 1 -- 172.21.15.136:0/475395724 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7fda080714f0 con 0x7fda180c2e00 2024-04-25T11:44:53.309 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 msgr2=0x7fd9f80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 0x7fd9f80719e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fda181553b0 tx=0x7fda0c006b50 comp rx=0 tx=0).stop 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 msgr2=0x7fda181f47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fda0801f680 tx=0x7fda0801fb40 comp rx=0 tx=0).stop 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 shutdown_connections 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda1815c1d0 0x7fda18147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda1815bdc0 0x7fda181e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd9f806f4d0 0x7fd9f80719e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 --2- 172.21.15.136:0/475395724 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda180c2e00 0x7fda181f47e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 >> 172.21.15.136:0/475395724 conn(0x7fda180ca5a0 msgr2=0x7fda180c38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 shutdown_connections 2024-04-25T11:44:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.324+0000 7fda04ff9640 1 -- 172.21.15.136:0/475395724 wait complete. 2024-04-25T11:44:53.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T11:44:53.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.760+0000 7f1a0e782640 1 Processor -- start 2024-04-25T11:44:53.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.760+0000 7f1a0e782640 1 -- start start 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 0x7f1a08073c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1a0815fe40 0x7f1a080c9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f1a08160250 con 0x7f1a0815fe40 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1a08074300 con 0x7f1a0806c670 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0e782640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a080745e0 con 0x7f1a080c9eb0 2024-04-25T11:44:53.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a07fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 0x7f1a08073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50186/0 (socket says 172.21.15.136:50186) 2024-04-25T11:44:53.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/1000958680 learned_addr learned my addr 172.21.15.136:0/1000958680 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:53.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a077fe640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.136:3300/0 conn(0x7f1a0815fe40 0x7f1a080c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/1000958680 >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 msgr2=0x7f1a08073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 0x7f1a08073c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/1000958680 >> v2:172.21.15.136:3300/0 conn(0x7f1a0815fe40 msgr2=0x7f1a080c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.136:3300/0 conn(0x7f1a0815fe40 0x7f1a080c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/1000958680 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a080748c0 con 0x7f1a080c9eb0 2024-04-25T11:44:53.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.764+0000 7f1a07fff640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 0x7f1a08073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/1000958680 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a080ce950 tx=0x7f1a0001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6acbb3bb1e6d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a06ffd640 1 -- 172.21.15.136:0/1000958680 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a000b4070 con 0x7f1a080c9eb0 2024-04-25T11:44:53.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a06ffd640 1 -- 172.21.15.136:0/1000958680 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a000af400 con 0x7f1a080c9eb0 2024-04-25T11:44:53.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a06ffd640 1 -- 172.21.15.136:0/1000958680 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a000af7d0 con 0x7f1a080c9eb0 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 -- 172.21.15.136:0/1000958680 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 msgr2=0x7f1a08073000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 --2- 172.21.15.136:0/1000958680 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a080ce950 tx=0x7f1a0001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 -- 172.21.15.136:0/1000958680 shutdown_connections 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.136:3300/0 conn(0x7f1a0815fe40 0x7f1a080c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 --2- 172.21.15.136:0/1000958680 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a08073000 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 --2- 172.21.15.136:0/1000958680 >> v2:172.21.15.157:3300/0 conn(0x7f1a0806c670 0x7f1a08073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.772+0000 7f1a0e782640 1 -- 172.21.15.136:0/1000958680 >> 172.21.15.136:0/1000958680 conn(0x7f1a080ca390 msgr2=0x7f1a080ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:53.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.776+0000 7f1a0e782640 1 -- 172.21.15.136:0/1000958680 shutdown_connections 2024-04-25T11:44:53.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.780+0000 7f1a0e782640 1 -- 172.21.15.136:0/1000958680 wait complete. 2024-04-25T11:44:53.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.788+0000 7f1a0e782640 1 Processor -- start 2024-04-25T11:44:53.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.788+0000 7f1a0e782640 1 -- start start 2024-04-25T11:44:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 0x7f1a081f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a081f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.778 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 0x7f1a080c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0815f920 con 0x7f1a0815fe40 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0815f640 con 0x7f1a0806c670 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0cd1f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 0x7f1a080c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 0x7f1a081f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a081f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a07fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 0x7f1a081f2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:37446/0 (socket says 172.21.15.136:37446) 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a077fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a081f5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:50198/0 (socket says 172.21.15.136:50198) 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a07fff640 1 -- 172.21.15.136:0/2945667314 learned_addr learned my addr 172.21.15.136:0/2945667314 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0e782640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a08073540 con 0x7f1a080c9eb0 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 msgr2=0x7f1a081f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a081f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 msgr2=0x7f1a081f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.792+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 0x7f1a081f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a0cd1f640 1 -- 172.21.15.136:0/2945667314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a081f2e40 con 0x7f1a0815fe40 2024-04-25T11:44:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a0cd1f640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 0x7f1a080c5dc0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a000ae000 tx=0x7f1a0001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a000b4040 con 0x7f1a0815fe40 2024-04-25T11:44:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a000afc90 con 0x7f1a0815fe40 2024-04-25T11:44:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a00009510 con 0x7f1a0815fe40 2024-04-25T11:44:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a0e782640 1 -- 172.21.15.136:0/2945667314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a081f64e0 con 0x7f1a0815fe40 2024-04-25T11:44:53.780 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a0e782640 1 -- 172.21.15.136:0/2945667314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a08156740 con 0x7f1a0815fe40 2024-04-25T11:44:53.781 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1a000c2060 con 0x7f1a0815fe40 2024-04-25T11:44:53.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 0x7f19ec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:53.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.796+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f1a0012cf10 con 0x7f1a0815fe40 2024-04-25T11:44:53.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.800+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e00052a0 con 0x7f1a0815fe40 2024-04-25T11:44:53.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.800+0000 7f1a07fff640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 0x7f19ec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:53.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.804+0000 7f1a07fff640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 0x7f19ec071940 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009600 tx=0x7f19f4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:53.797 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:53.808+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1a00018d40 con 0x7f1a0815fe40 2024-04-25T11:44:54.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.312+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19e0005da0 con 0x7f1a0815fe40 2024-04-25T11:44:54.296 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.312+0000 7f1a04ff9640 1 -- 172.21.15.136:0/2945667314 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) ==== 104+0+11 (secure 0 0 0) 0x7f1a000f9680 con 0x7f1a0815fe40 2024-04-25T11:44:54.297 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 msgr2=0x7f19ec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 0x7f19ec071940 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009600 tx=0x7f19f4009190 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 msgr2=0x7f1a080c5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 0x7f1a080c5dc0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a000ae000 tx=0x7f1a0001bf80 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 shutdown_connections 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a0815fe40 0x7f1a080c5dc0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f19ec06f430 0x7f19ec071940 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a080c9eb0 0x7f1a081f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 --2- 172.21.15.136:0/2945667314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a0806c670 0x7f1a081f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 >> 172.21.15.136:0/2945667314 conn(0x7f1a080ca390 msgr2=0x7f1a080ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:54.299 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 shutdown_connections 2024-04-25T11:44:54.300 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.316+0000 7f19f27fc640 1 -- 172.21.15.136:0/2945667314 wait complete. 2024-04-25T11:44:54.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T11:44:54.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.796+0000 7fb714f60640 1 Processor -- start 2024-04-25T11:44:54.782 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.796+0000 7fb714f60640 1 -- start start 2024-04-25T11:44:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb714f60640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 0x7fb71015a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb714f60640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb7100c7c50 0x7fb71006fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb714f60640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7100729c0 con 0x7fb7100c8060 2024-04-25T11:44:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb714f60640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fb710070940 con 0x7fb7101603a0 2024-04-25T11:44:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb714f60640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb710074300 con 0x7fb7100c7c50 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb7100c7c50 0x7fb71006fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb7100c7c50 0x7fb71006fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52408/0 (socket says 172.21.15.136:52408) 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 -- 172.21.15.136:0/173046156 learned_addr learned my addr 172.21.15.136:0/173046156 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70e59c640 1 --2- 172.21.15.136:0/173046156 >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 0x7fb71015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 -- 172.21.15.136:0/173046156 >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 msgr2=0x7fb71015a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70d59a640 1 -- 172.21.15.136:0/173046156 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491101497 0 0) 0x7fb7100729c0 con 0x7fb7100c8060 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 --2- 172.21.15.136:0/173046156 >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 0x7fb71015a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 -- 172.21.15.136:0/173046156 >> v1:172.21.15.157:6789/0 conn(0x7fb7100c8060 legacy=0x7fb710070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.804+0000 7fb70dd9b640 1 -- 172.21.15.136:0/173046156 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb710072c80 con 0x7fb7100c7c50 2024-04-25T11:44:54.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb70e59c640 1 --2- 172.21.15.136:0/173046156 >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 0x7fb71015a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:54.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb70dd9b640 1 --2- 172.21.15.136:0/173046156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c7c50 0x7fb71006fcf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb7100720f0 tx=0x7fb70401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3bc5cf3d83e25d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:54.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb70d59a640 1 -- 172.21.15.136:0/173046156 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7040b5070 con 0x7fb7100c7c50 2024-04-25T11:44:54.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb70d59a640 1 -- 172.21.15.136:0/173046156 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7040a8a10 con 0x7fb7100c7c50 2024-04-25T11:44:54.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb70d59a640 1 -- 172.21.15.136:0/173046156 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7040a8de0 con 0x7fb7100c7c50 2024-04-25T11:44:54.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb714f60640 1 -- 172.21.15.136:0/173046156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c7c50 msgr2=0x7fb71006fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.808+0000 7fb714f60640 1 --2- 172.21.15.136:0/173046156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c7c50 0x7fb71006fcf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb7100720f0 tx=0x7fb70401ed30 comp rx=0 tx=0).stop 2024-04-25T11:44:54.800 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 -- 172.21.15.136:0/173046156 shutdown_connections 2024-04-25T11:44:54.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 --2- 172.21.15.136:0/173046156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c7c50 0x7fb71006fcf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 --2- 172.21.15.136:0/173046156 >> v2:172.21.15.136:3300/0 conn(0x7fb7101603a0 0x7fb71015a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 -- 172.21.15.136:0/173046156 >> 172.21.15.136:0/173046156 conn(0x7fb7100c9d50 msgr2=0x7fb7100ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:54.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 -- 172.21.15.136:0/173046156 shutdown_connections 2024-04-25T11:44:54.801 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.816+0000 7fb714f60640 1 -- 172.21.15.136:0/173046156 wait complete. 2024-04-25T11:44:54.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.820+0000 7fb714f60640 1 Processor -- start 2024-04-25T11:44:54.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.820+0000 7fb714f60640 1 -- start start 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.820+0000 7fb714f60640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 0x7fb710155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.820+0000 7fb714f60640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 0x7fb710147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.820+0000 7fb714f60640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43954/0 (socket says 172.21.15.136:43954) 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 -- 172.21.15.136:0/4182828582 learned_addr learned my addr 172.21.15.136:0/4182828582 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb710073520 con 0x7fb7101603a0 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb710073240 con 0x7fb7100c8060 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70e59c640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 0x7fb710155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb710073800 con 0x7fb7100c7c50 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70dd9b640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 0x7fb710147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.808 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 -- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 msgr2=0x7fb710155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 0x7fb710155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 -- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 msgr2=0x7fb710147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 0x7fb710147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb710147ba0 con 0x7fb7101603a0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70e59c640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 0x7fb710155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70dd9b640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 0x7fb710147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb70ed9d640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c700 tx=0x7fb70800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708017070 con 0x7fb7101603a0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb708003db0 con 0x7fb7101603a0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708005310 con 0x7fb7101603a0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7101f7220 con 0x7fb7101603a0 2024-04-25T11:44:54.809 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7101eb570 con 0x7fb7101603a0 2024-04-25T11:44:54.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71006a350 con 0x7fb7101603a0 2024-04-25T11:44:54.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.824+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb70801a070 con 0x7fb7101603a0 2024-04-25T11:44:54.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.828+0000 7fb6feffd640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 0x7fb6f0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:54.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.828+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fb70808efe0 con 0x7fb7101603a0 2024-04-25T11:44:54.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.828+0000 7fb70e59c640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 0x7fb6f0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:54.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.828+0000 7fb70e59c640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 0x7fb6f0071940 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb700009580 tx=0x7fb700009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:54.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:54.836+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb70805b0e0 con 0x7fb7101603a0 2024-04-25T11:44:55.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.164+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7100c4750 con 0x7fb7101603a0 2024-04-25T11:44:55.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.164+0000 7fb6feffd640 1 -- 172.21.15.136:0/4182828582 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7fb70805f3e0 con 0x7fb7101603a0 2024-04-25T11:44:55.149 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 64 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 msgr2=0x7fb6f0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 0x7fb6f0071940 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb700009580 tx=0x7fb700009190 comp rx=0 tx=0).stop 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 msgr2=0x7fb710147660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c700 tx=0x7fb70800cbc0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 shutdown_connections 2024-04-25T11:44:55.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101603a0 0x7fb710147660 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7100c8060 0x7fb710147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fb6f006f430 0x7fb6f0071940 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 --2- 172.21.15.136:0/4182828582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7100c7c50 0x7fb710155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 >> 172.21.15.136:0/4182828582 conn(0x7fb7100c9d50 msgr2=0x7fb71015dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 shutdown_connections 2024-04-25T11:44:55.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.168+0000 7fb714f60640 1 -- 172.21.15.136:0/4182828582 wait complete. 2024-04-25T11:44:55.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T11:44:55.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 Processor -- start 2024-04-25T11:44:55.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 -- start start 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9a780c7c50 0x7f9a7806fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a780729c0 con 0x7f9a781603a0 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f9a78074300 con 0x7f9a780c7c50 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7d1b7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a78072c80 con 0x7f9a780c8060 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:44374/0 (socket says 172.21.15.136:44374) 2024-04-25T11:44:55.530 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 -- 172.21.15.136:0/1737923086 learned_addr learned my addr 172.21.15.136:0/1737923086 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:55.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 -- 172.21.15.136:0/1737923086 >> v2:172.21.15.136:3300/0 conn(0x7f9a780c7c50 msgr2=0x7f9a7806fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:55.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 --2- 172.21.15.136:0/1737923086 >> v2:172.21.15.136:3300/0 conn(0x7f9a780c7c50 0x7f9a7806fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 -- 172.21.15.136:0/1737923086 >> v1:172.21.15.157:6789/0 conn(0x7f9a781603a0 legacy=0x7f9a7815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:55.531 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.544+0000 7f9a7759e640 1 -- 172.21.15.136:0/1737923086 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a78072f60 con 0x7f9a780c8060 2024-04-25T11:44:55.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7759e640 1 --2- 172.21.15.136:0/1737923086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009e50 tx=0x7f9a7001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83bddc6e3cdbab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:55.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7659c640 1 -- 172.21.15.136:0/1737923086 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700b4070 con 0x7f9a780c8060 2024-04-25T11:44:55.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7659c640 1 -- 172.21.15.136:0/1737923086 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a700af400 con 0x7f9a780c8060 2024-04-25T11:44:55.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7659c640 1 -- 172.21.15.136:0/1737923086 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a700af7d0 con 0x7f9a780c8060 2024-04-25T11:44:55.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1737923086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 msgr2=0x7f9a78070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1737923086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009e50 tx=0x7f9a7001f1e0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.536 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1737923086 shutdown_connections 2024-04-25T11:44:55.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1737923086 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a780c8060 0x7f9a78070230 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1737923086 >> v2:172.21.15.136:3300/0 conn(0x7f9a780c7c50 0x7f9a7806fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.552+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1737923086 >> 172.21.15.136:0/1737923086 conn(0x7f9a780c9d50 msgr2=0x7f9a780ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:55.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.568+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1737923086 shutdown_connections 2024-04-25T11:44:55.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.572+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1737923086 wait complete. 2024-04-25T11:44:55.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.576+0000 7f9a7d1b7640 1 Processor -- start 2024-04-25T11:44:55.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.576+0000 7f9a7d1b7640 1 -- start start 2024-04-25T11:44:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a780c7c50 0x7f9a781f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a781603a0 0x7f9a780c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78073800 con 0x7f9a780c7c50 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9a78073520 con 0x7f9a780c8060 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7d1b7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9a78073d20 con 0x7f9a781603a0 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52422/0 (socket says 172.21.15.136:52422) 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 -- 172.21.15.136:0/1747159433 learned_addr learned my addr 172.21.15.136:0/1747159433 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a7759e640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a781603a0 0x7f9a780c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:55.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a76d9d640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a780c7c50 0x7f9a781f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 -- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a781603a0 msgr2=0x7f9a780c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a781603a0 0x7f9a780c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 -- 172.21.15.136:0/1747159433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a780c7c50 msgr2=0x7f9a781f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a780c7c50 0x7f9a781f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 -- 172.21.15.136:0/1747159433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a781f6880 con 0x7f9a780c8060 2024-04-25T11:44:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.580+0000 7f9a6e59c640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b450 tx=0x7f9a6800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:55.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68015020 con 0x7f9a780c8060 2024-04-25T11:44:55.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a68011890 con 0x7f9a780c8060 2024-04-25T11:44:55.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68011c80 con 0x7f9a780c8060 2024-04-25T11:44:55.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a780c70c0 con 0x7f9a780c8060 2024-04-25T11:44:55.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a781f6af0 con 0x7f9a780c8060 2024-04-25T11:44:55.569 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c0052a0 con 0x7f9a780c8060 2024-04-25T11:44:55.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.584+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a68018020 con 0x7f9a780c8060 2024-04-25T11:44:55.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.592+0000 7f9a6ffff640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 0x7f9a64071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:55.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.592+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f9a6808fba0 con 0x7f9a780c8060 2024-04-25T11:44:55.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.592+0000 7f9a76d9d640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 0x7f9a64071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:55.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.592+0000 7f9a76d9d640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 0x7f9a64071aa0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000b810 tx=0x7f9a60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:55.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.596+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a6801f020 con 0x7f9a780c8060 2024-04-25T11:44:55.898 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.916+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9a5c005da0 con 0x7f9a780c8060 2024-04-25T11:44:55.899 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.916+0000 7f9a6ffff640 1 -- 172.21.15.136:0/1747159433 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v40) ==== 103+0+11 (secure 0 0 0) 0x7f9a6800d040 con 0x7f9a780c8060 2024-04-25T11:44:55.899 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 32 2024-04-25T11:44:55.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.916+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 msgr2=0x7f9a64071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.916+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 0x7f9a64071aa0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000b810 tx=0x7f9a60008040 comp rx=0 tx=0).stop 2024-04-25T11:44:55.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.916+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 msgr2=0x7f9a780c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:55.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800b450 tx=0x7f9a6800b910 comp rx=0 tx=0).stop 2024-04-25T11:44:55.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 shutdown_connections 2024-04-25T11:44:55.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f9a6406f590 0x7f9a64071aa0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9a781603a0 0x7f9a780c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a780c8060 0x7f9a780c6980 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 --2- 172.21.15.136:0/1747159433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a780c7c50 0x7f9a781f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:55.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.920+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 >> 172.21.15.136:0/1747159433 conn(0x7f9a780c9d50 msgr2=0x7f9a7814f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:55.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.924+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 shutdown_connections 2024-04-25T11:44:55.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:55.924+0000 7f9a7d1b7640 1 -- 172.21.15.136:0/1747159433 wait complete. 2024-04-25T11:44:55.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T11:44:56.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.204+0000 7fbf94d8b640 1 Processor -- start 2024-04-25T11:44:56.190 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.204+0000 7fbf94d8b640 1 -- start start 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf94d8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf900ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf94d8b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf900ca010 0x7fbf900ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf94d8b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900cbad0 con 0x7fbf900c8cd0 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf94d8b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbf900c8550 con 0x7fbf900ca010 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf94d8b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbf900c7c20 con 0x7fbf900ce260 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf900ca010 0x7fbf900ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbf900ca010 0x7fbf900ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52424/0 (socket says 172.21.15.136:52424) 2024-04-25T11:44:56.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/362369419 learned_addr learned my addr 172.21.15.136:0/362369419 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 msgr2=0x7fbf900ce670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T11:44:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf900ce670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/362369419 >> v1:172.21.15.136:6789/0 conn(0x7fbf900c8cd0 legacy=0x7fbf9014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:44:56.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.208+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/362369419 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf900c7f00 con 0x7fbf900ca010 2024-04-25T11:44:56.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.212+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900ca010 0x7fbf900ca420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf900c86d0 tx=0x7fbf8801f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f20c2fb3b3f26ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:56.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.212+0000 7fbf8dd9b640 1 -- 172.21.15.136:0/362369419 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf880b4070 con 0x7fbf900ca010 2024-04-25T11:44:56.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.212+0000 7fbf8dd9b640 1 -- 172.21.15.136:0/362369419 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf880af400 con 0x7fbf900ca010 2024-04-25T11:44:56.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.212+0000 7fbf8dd9b640 1 -- 172.21.15.136:0/362369419 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf880af7d0 con 0x7fbf900ca010 2024-04-25T11:44:56.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 -- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900ca010 msgr2=0x7fbf900ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:56.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900ca010 0x7fbf900ca420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbf900c86d0 tx=0x7fbf8801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 -- 172.21.15.136:0/362369419 shutdown_connections 2024-04-25T11:44:56.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900ca010 0x7fbf900ca420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/362369419 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf900ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.200 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.216+0000 7fbf94d8b640 1 -- 172.21.15.136:0/362369419 >> 172.21.15.136:0/362369419 conn(0x7fbf9006f9d0 msgr2=0x7fbf9006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:56.209 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.224+0000 7fbf94d8b640 1 -- 172.21.15.136:0/362369419 shutdown_connections 2024-04-25T11:44:56.214 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.228+0000 7fbf94d8b640 1 -- 172.21.15.136:0/362369419 wait complete. 2024-04-25T11:44:56.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.232+0000 7fbf94d8b640 1 Processor -- start 2024-04-25T11:44:56.218 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.232+0000 7fbf94d8b640 1 -- start start 2024-04-25T11:44:56.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 0x7fbf901f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900ca010 0x7fbf90153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf90148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbf900cb720 con 0x7fbf900ca010 2024-04-25T11:44:56.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 0x7fbf901f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8e59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 0x7fbf901f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:52436/0 (socket says 172.21.15.136:52436) 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8e59c640 1 -- 172.21.15.136:0/3810178899 learned_addr learned my addr 172.21.15.136:0/3810178899 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf90148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbf900ca960 con 0x7fbf900c8cd0 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbf9014a4b0 con 0x7fbf900ce260 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 msgr2=0x7fbf901f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:56.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 0x7fbf901f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/3810178899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900ca010 msgr2=0x7fbf90153930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T11:44:56.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900ca010 0x7fbf90153930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9014df50 con 0x7fbf900ce260 2024-04-25T11:44:56.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.240+0000 7fbf8ed9d640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf90148890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf880ae000 tx=0x7fbf8801bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:56.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.244+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf880b4040 con 0x7fbf900ce260 2024-04-25T11:44:56.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.244+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9015bca0 con 0x7fbf900ce260 2024-04-25T11:44:56.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.244+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf901f2ce0 con 0x7fbf900ce260 2024-04-25T11:44:56.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.244+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf880af6f0 con 0x7fbf900ce260 2024-04-25T11:44:56.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.248+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf880afae0 con 0x7fbf900ce260 2024-04-25T11:44:56.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.248+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9006a350 con 0x7fbf900ce260 2024-04-25T11:44:56.232 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.248+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbf880c6020 con 0x7fbf900ce260 2024-04-25T11:44:56.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.248+0000 7fbf86ffd640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 0x7fbf70071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:44:56.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.248+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fbf88130aa0 con 0x7fbf900ce260 2024-04-25T11:44:56.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.252+0000 7fbf8e59c640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 0x7fbf70071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:44:56.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.252+0000 7fbf8e59c640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 0x7fbf70071960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbf800098b0 tx=0x7fbf80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:44:56.239 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.256+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbf880cd020 con 0x7fbf900ce260 2024-04-25T11:44:56.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.636+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbf9015ba00 con 0x7fbf900ce260 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.640+0000 7fbf86ffd640 1 -- 172.21.15.136:0/3810178899 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 28 entries v40) ==== 81+0+1728 (secure 0 0 0) 0x7fbf88018d40 con 0x7fbf900ce260 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/1501623433 2024-04-26T11:41:59.046316+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/1501623433 2024-04-26T11:41:59.046316+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/3576995854 2024-04-26T11:36:58.974013+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/3576995854 2024-04-26T11:36:58.974013+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/3688751078 2024-04-26T11:33:28.923953+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6836/3463942074 2024-04-26T11:38:18.992149+0000 2024-04-25T11:44:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6834/430807169 2024-04-26T11:23:23.787444+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6838/3020784477 2024-04-26T11:42:54.061351+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6835/430807169 2024-04-26T11:23:23.787444+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6839/3020784477 2024-04-26T11:42:54.061351+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6833/3353376405 2024-04-26T11:25:58.820255+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6836/1740633057 2024-04-26T11:44:19.077214+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6832/906354453 2024-04-26T11:39:24.008268+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6834/3463942074 2024-04-26T11:38:18.992149+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6832/3353376405 2024-04-26T11:25:58.820255+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6839/1671830331 2024-04-26T11:32:38.912872+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6834/3406304193 2024-04-26T11:35:58.955925+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6836/1368493675 2024-04-26T11:28:33.853864+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6837/1740633057 2024-04-26T11:44:19.077214+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6838/1671830331 2024-04-26T11:32:38.912872+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6837/1368493675 2024-04-26T11:28:33.853864+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/3688751078 2024-04-26T11:33:28.923953+0000 2024-04-25T11:44:56.625 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6835/281980057 2024-04-26T11:30:08.877388+0000 2024-04-25T11:44:56.626 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6833/906354453 2024-04-26T11:39:24.008268+0000 2024-04-25T11:44:56.626 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6837/281980057 2024-04-26T11:30:08.877388+0000 2024-04-25T11:44:56.626 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136:6835/3406304193 2024-04-26T11:35:58.955925+0000 2024-04-25T11:44:56.626 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6834/694883637 2024-04-26T11:30:48.887575+0000 2024-04-25T11:44:56.626 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.157:6836/694883637 2024-04-26T11:30:48.887575+0000 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:listed 28 entries 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.640+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 msgr2=0x7fbf70071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.640+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 0x7fbf70071960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbf800098b0 tx=0x7fbf80008040 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 msgr2=0x7fbf90148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf90148890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf880ae000 tx=0x7fbf8801bf80 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 shutdown_connections 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fbf7006f450 0x7fbf70071960 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbf900ce260 0x7fbf90148890 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900ca010 0x7fbf90153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 --2- 172.21.15.136:0/3810178899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbf900c8cd0 0x7fbf901f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:44:56.627 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 >> 172.21.15.136:0/3810178899 conn(0x7fbf9006f9d0 msgr2=0x7fbf90075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:44:56.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 shutdown_connections 2024-04-25T11:44:56.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:44:56.644+0000 7fbf94d8b640 1 -- 172.21.15.136:0/3810178899 wait complete. 2024-04-25T11:44:56.664 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi136.front.sepia.ceph.com... 2024-04-25T11:44:56.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:44:56.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T11:44:56.725 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:44:56.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi136.stderr:ceph-fuse[35981]: fuse finished with error 0 and tester_r 0 2024-04-25T11:45:02.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:45:02.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:02.827 DEBUG:teuthology.orchestra.run.smithi136:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T11:45:02.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:02.838 DEBUG:teuthology.orchestra.run.smithi136:> ip netns list 2024-04-25T11:45:03.009 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T11:45:03.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:03.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T11:45:03.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T11:45:03.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo ip link delete ceph-brx 2024-04-25T11:45:03.180 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T11:45:03.190 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T11:45:03.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:03.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.347+0000 7ff1ff2dc640 1 Processor -- start 2024-04-25T11:45:03.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.347+0000 7ff1ff2dc640 1 -- start start 2024-04-25T11:45:03.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1ff2dc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1f80c96e0 0x7ff1f81540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1ff2dc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f80c9af0 con 0x7ff1f815aae0 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1ff2dc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f80cb220 con 0x7ff1f80c79e0 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1ff2dc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f806cc10 con 0x7ff1f80c96e0 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd879640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff1f815aae0 0x7ff1f8164900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55546/0 (socket says 172.21.15.136:55546) 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd879640 1 -- 172.21.15.136:0/1651572672 learned_addr learned my addr 172.21.15.136:0/1651572672 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd078640 1 --2- 172.21.15.136:0/1651572672 >> v2:172.21.15.136:3300/0 conn(0x7ff1f80c96e0 0x7ff1f81540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd078640 1 -- 172.21.15.136:0/1651572672 >> v1:172.21.15.157:6789/0 conn(0x7ff1f80c79e0 legacy=0x7ff1f815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd078640 1 -- 172.21.15.136:0/1651572672 >> v1:172.21.15.136:6789/0 conn(0x7ff1f815aae0 legacy=0x7ff1f8164900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1effff640 1 -- 172.21.15.136:0/1651572672 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112543697 0 0) 0x7ff1f80c9af0 con 0x7ff1f815aae0 2024-04-25T11:45:03.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd078640 1 -- 172.21.15.136:0/1651572672 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f80cb500 con 0x7ff1f80c96e0 2024-04-25T11:45:03.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.351+0000 7ff1fd078640 1 --2- 172.21.15.136:0/1651572672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81540d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff1f806cd90 tx=0x7ff1f401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b82ba2bb915d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1effff640 1 -- 172.21.15.136:0/1651572672 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40b5070 con 0x7ff1f80c96e0 2024-04-25T11:45:03.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1effff640 1 -- 172.21.15.136:0/1651572672 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f40a8950 con 0x7ff1f80c96e0 2024-04-25T11:45:03.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1effff640 1 -- 172.21.15.136:0/1651572672 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40a8d20 con 0x7ff1f80c96e0 2024-04-25T11:45:03.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1651572672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 msgr2=0x7ff1f81540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1651572672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81540d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ff1f806cd90 tx=0x7ff1f401ec50 comp rx=0 tx=0).stop 2024-04-25T11:45:03.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1651572672 shutdown_connections 2024-04-25T11:45:03.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1651572672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81540d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1651572672 >> 172.21.15.136:0/1651572672 conn(0x7ff1f806f9d0 msgr2=0x7ff1f806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:03.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1651572672 shutdown_connections 2024-04-25T11:45:03.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1651572672 wait complete. 2024-04-25T11:45:03.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 Processor -- start 2024-04-25T11:45:03.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- start start 2024-04-25T11:45:03.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80c79e0 0x7ff1f814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815aae0 0x7ff1f8154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f80cf0f0 con 0x7ff1f80c96e0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f8154610 con 0x7ff1f815aae0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.355+0000 7ff1ff2dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f80ced40 con 0x7ff1f80c79e0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:43996/0 (socket says 172.21.15.136:43996) 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 -- 172.21.15.136:0/1333570385 learned_addr learned my addr 172.21.15.136:0/1333570385 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fd879640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815aae0 0x7ff1f8154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fd078640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80c79e0 0x7ff1f814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 -- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80c79e0 msgr2=0x7ff1f814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80c79e0 0x7ff1f814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 -- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815aae0 msgr2=0x7ff1f8154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815aae0 0x7ff1f8154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 -- 172.21.15.136:0/1333570385 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f81f68c0 con 0x7ff1f80c96e0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fc877640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b440 tx=0x7ff1e800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8015020 con 0x7ff1f80c96e0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e8011890 con 0x7ff1f80c96e0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f814e600 con 0x7ff1f80c96e0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8011c80 con 0x7ff1f80c96e0 2024-04-25T11:45:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f81f27a0 con 0x7ff1f80c96e0 2024-04-25T11:45:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f814e8f0 con 0x7ff1f80c96e0 2024-04-25T11:45:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1e8018020 con 0x7ff1f80c96e0 2024-04-25T11:45:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 0x7ff1dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7ff1e808fcf0 con 0x7ff1f80c96e0 2024-04-25T11:45:03.346 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.359+0000 7ff1fd078640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 0x7ff1dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.363+0000 7ff1fd078640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 0x7ff1dc071940 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40b2000 tx=0x7ff1f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.363+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff1e801f030 con 0x7ff1f80c96e0 2024-04-25T11:45:03.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.515+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1f806a350 con 0x7ff1dc06f430 2024-04-25T11:45:03.529 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.543+0000 7ff1edffb640 1 -- 172.21.15.136:0/1333570385 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334465 (secure 0 0 0) 0x7ff1f806a350 con 0x7ff1dc06f430 2024-04-25T11:45:03.533 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:03.536 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 msgr2=0x7ff1dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 0x7ff1dc071940 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40b2000 tx=0x7ff1f4008040 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 msgr2=0x7ff1f81f6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b440 tx=0x7ff1e800b900 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 shutdown_connections 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815aae0 0x7ff1f8154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80c96e0 0x7ff1f81f6380 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1dc06f430 0x7ff1dc071940 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 --2- 172.21.15.136:0/1333570385 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80c79e0 0x7ff1f814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 >> 172.21.15.136:0/1333570385 conn(0x7ff1f806f9d0 msgr2=0x7ff1f8152650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:03.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 shutdown_connections 2024-04-25T11:45:03.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.547+0000 7ff1ff2dc640 1 -- 172.21.15.136:0/1333570385 wait complete. 2024-04-25T11:45:03.553 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-04-25T11:45:03.415956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642572045,"num_objects":385,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658150,"num_write_kb":15736462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160269,"ondisk_log_size":160269,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":10443404,"kb_used_data":1543564,"kb_used_omap":12,"kb_used_meta":8893619,"kb_avail":744531316,"statfs":{"total":773094113280,"available":762400067584,"internally_reserved":0,"allocated":1580609536,"data_stored":1578527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9107066418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,13,8,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8339203214,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2050,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003730"},"pg_stats":[{"pgid":"2.1f","version":"40'272","reported_seq":358,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.557577+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:44:53.557577+0000","last_peered":"2024-04-25T11:44:53.557577+0000","last_clean":"2024-04-25T11:44:53.557577+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:44:53.557577+0000","last_undegraded":"2024-04-25T11:44:53.557577+0000","last_fullsized":"2024-04-25T11:44:53.557577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'4","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588204+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:44:28.588204+0000","last_peered":"2024-04-25T11:44:28.588204+0000","last_clean":"2024-04-25T11:44:28.588204+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:44:28.588204+0000","last_undegraded":"2024-04-25T11:44:28.588204+0000","last_fullsized":"2024-04-25T11:44:28.588204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5319","reported_seq":5424,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.505149+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:44:58.505149+0000","last_peered":"2024-04-25T11:44:58.505149+0000","last_clean":"2024-04-25T11:44:58.505149+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:44:58.505149+0000","last_undegraded":"2024-04-25T11:44:58.505149+0000","last_fullsized":"2024-04-25T11:44:58.505149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67156925,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22037,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'9","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588290+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:44:28.588290+0000","last_peered":"2024-04-25T11:44:28.588290+0000","last_clean":"2024-04-25T11:44:28.588290+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:44:28.588290+0000","last_undegraded":"2024-04-25T11:44:28.588290+0000","last_fullsized":"2024-04-25T11:44:28.588290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.171897+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:44:47.171897+0000","last_peered":"2024-04-25T11:44:47.171897+0000","last_clean":"2024-04-25T11:44:47.171897+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:44:47.171897+0000","last_undegraded":"2024-04-25T11:44:47.171897+0000","last_fullsized":"2024-04-25T11:44:47.171897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4852","reported_seq":4956,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.839347+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:45:00.839347+0000","last_peered":"2024-04-25T11:45:00.839347+0000","last_clean":"2024-04-25T11:45:00.839347+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:00.839347+0000","last_undegraded":"2024-04-25T11:45:00.839347+0000","last_fullsized":"2024-04-25T11:45:00.839347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35243,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20160,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'28","reported_seq":122,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.650684+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:44:42.650684+0000","last_peered":"2024-04-25T11:44:42.650684+0000","last_clean":"2024-04-25T11:44:42.650684+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:42.650684+0000","last_undegraded":"2024-04-25T11:44:42.650684+0000","last_fullsized":"2024-04-25T11:44:42.650684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.299464+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:44:50.299464+0000","last_peered":"2024-04-25T11:44:50.299464+0000","last_clean":"2024-04-25T11:44:50.299464+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:44:50.299464+0000","last_undegraded":"2024-04-25T11:44:50.299464+0000","last_fullsized":"2024-04-25T11:44:50.299464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4262","reported_seq":4357,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.572827+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:45:01.572827+0000","last_peered":"2024-04-25T11:45:01.572827+0000","last_clean":"2024-04-25T11:45:01.572827+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:01.572827+0000","last_undegraded":"2024-04-25T11:45:01.572827+0000","last_fullsized":"2024-04-25T11:45:01.572827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17219,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108293+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:44:19.108293+0000","last_peered":"2024-04-25T11:44:19.108293+0000","last_clean":"2024-04-25T11:44:19.108293+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:44:19.108293+0000","last_undegraded":"2024-04-25T11:44:19.108293+0000","last_fullsized":"2024-04-25T11:44:19.108293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":62,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.649419+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:45:01.649419+0000","last_peered":"2024-04-25T11:45:01.649419+0000","last_clean":"2024-04-25T11:45:01.649419+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:01.649419+0000","last_undegraded":"2024-04-25T11:45:01.649419+0000","last_fullsized":"2024-04-25T11:45:01.649419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4167","reported_seq":4270,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.384290+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:45:01.384290+0000","last_peered":"2024-04-25T11:45:01.384290+0000","last_clean":"2024-04-25T11:45:01.384290+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:01.384290+0000","last_undegraded":"2024-04-25T11:45:01.384290+0000","last_fullsized":"2024-04-25T11:45:01.384290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16706,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"26'6","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.110341+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:44:19.110341+0000","last_peered":"2024-04-25T11:44:19.110341+0000","last_clean":"2024-04-25T11:44:19.110341+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:44:19.110341+0000","last_undegraded":"2024-04-25T11:44:19.110341+0000","last_fullsized":"2024-04-25T11:44:19.110341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1087","reported_seq":1166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105876+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:44:19.105876+0000","last_peered":"2024-04-25T11:44:19.105876+0000","last_clean":"2024-04-25T11:44:19.105876+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:44:19.105876+0000","last_undegraded":"2024-04-25T11:44:19.105876+0000","last_fullsized":"2024-04-25T11:44:19.105876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5157","reported_seq":5255,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482495+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:45:01.482495+0000","last_peered":"2024-04-25T11:45:01.482495+0000","last_clean":"2024-04-25T11:45:01.482495+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:01.482495+0000","last_undegraded":"2024-04-25T11:45:01.482495+0000","last_fullsized":"2024-04-25T11:45:01.482495+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2946,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21466,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"26'5","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.134609+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:44:19.134609+0000","last_peered":"2024-04-25T11:44:19.134609+0000","last_clean":"2024-04-25T11:44:19.134609+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:44:19.134609+0000","last_undegraded":"2024-04-25T11:44:19.134609+0000","last_fullsized":"2024-04-25T11:44:19.134609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"39'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.124512+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:44:19.124512+0000","last_peered":"2024-04-25T11:44:19.124512+0000","last_clean":"2024-04-25T11:44:19.124512+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:44:19.124512+0000","last_undegraded":"2024-04-25T11:44:19.124512+0000","last_fullsized":"2024-04-25T11:44:19.124512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5165","reported_seq":5262,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.463179+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:45:01.463179+0000","last_peered":"2024-04-25T11:45:01.463179+0000","last_clean":"2024-04-25T11:45:01.463179+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:01.463179+0000","last_undegraded":"2024-04-25T11:45:01.463179+0000","last_fullsized":"2024-04-25T11:45:01.463179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15573,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21560,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.325387+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:44:50.325387+0000","last_peered":"2024-04-25T11:44:50.325387+0000","last_clean":"2024-04-25T11:44:50.325387+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:44:50.325387+0000","last_undegraded":"2024-04-25T11:44:50.325387+0000","last_fullsized":"2024-04-25T11:44:50.325387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111004+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:44:19.111004+0000","last_peered":"2024-04-25T11:44:19.111004+0000","last_clean":"2024-04-25T11:44:19.111004+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:44:19.111004+0000","last_undegraded":"2024-04-25T11:44:19.111004+0000","last_fullsized":"2024-04-25T11:44:19.111004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6708,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.767277+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:45:01.767277+0000","last_peered":"2024-04-25T11:45:01.767277+0000","last_clean":"2024-04-25T11:45:01.767277+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:01.767277+0000","last_undegraded":"2024-04-25T11:45:01.767277+0000","last_fullsized":"2024-04-25T11:45:01.767277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.688232+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:44:58.688232+0000","last_peered":"2024-04-25T11:44:58.688232+0000","last_clean":"2024-04-25T11:44:58.688232+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:44:58.688232+0000","last_undegraded":"2024-04-25T11:44:58.688232+0000","last_fullsized":"2024-04-25T11:44:58.688232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'6","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105377+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:44:19.105377+0000","last_peered":"2024-04-25T11:44:19.105377+0000","last_clean":"2024-04-25T11:44:19.105377+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:44:19.105377+0000","last_undegraded":"2024-04-25T11:44:19.105377+0000","last_fullsized":"2024-04-25T11:44:19.105377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4791","reported_seq":4887,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.422519+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:45:01.422519+0000","last_peered":"2024-04-25T11:45:01.422519+0000","last_clean":"2024-04-25T11:45:01.422519+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:01.422519+0000","last_undegraded":"2024-04-25T11:45:01.422519+0000","last_fullsized":"2024-04-25T11:45:01.422519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19671,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'17","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.332441+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:44:50.332441+0000","last_peered":"2024-04-25T11:44:50.332441+0000","last_clean":"2024-04-25T11:44:50.332441+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:44:50.332441+0000","last_undegraded":"2024-04-25T11:44:50.332441+0000","last_fullsized":"2024-04-25T11:44:50.332441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'10","reported_seq":97,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111925+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:44:19.111925+0000","last_peered":"2024-04-25T11:44:19.111925+0000","last_clean":"2024-04-25T11:44:19.111925+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:44:19.111925+0000","last_undegraded":"2024-04-25T11:44:19.111925+0000","last_fullsized":"2024-04-25T11:44:19.111925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4481","reported_seq":4582,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.591684+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:45:01.591684+0000","last_peered":"2024-04-25T11:45:01.591684+0000","last_clean":"2024-04-25T11:45:01.591684+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:01.591684+0000","last_undegraded":"2024-04-25T11:45:01.591684+0000","last_fullsized":"2024-04-25T11:45:01.591684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18512,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"26'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.174729+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:44:19.174729+0000","last_peered":"2024-04-25T11:44:19.174729+0000","last_clean":"2024-04-25T11:44:19.174729+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:44:19.174729+0000","last_undegraded":"2024-04-25T11:44:19.174729+0000","last_fullsized":"2024-04-25T11:44:19.174729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'367","reported_seq":447,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.589654+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:44:48.589654+0000","last_peered":"2024-04-25T11:44:48.589654+0000","last_clean":"2024-04-25T11:44:48.589654+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:44:48.589654+0000","last_undegraded":"2024-04-25T11:44:48.589654+0000","last_fullsized":"2024-04-25T11:44:48.589654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4523","reported_seq":4623,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.545359+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:44:58.545359+0000","last_peered":"2024-04-25T11:44:58.545359+0000","last_clean":"2024-04-25T11:44:58.545359+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:44:58.545359+0000","last_undegraded":"2024-04-25T11:44:58.545359+0000","last_fullsized":"2024-04-25T11:44:58.545359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58795404,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19016,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'7","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.341334+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:44:47.341334+0000","last_peered":"2024-04-25T11:44:47.341334+0000","last_clean":"2024-04-25T11:44:47.341334+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:44:47.341334+0000","last_undegraded":"2024-04-25T11:44:47.341334+0000","last_fullsized":"2024-04-25T11:44:47.341334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'6","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.106657+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:44:19.106657+0000","last_peered":"2024-04-25T11:44:19.106657+0000","last_clean":"2024-04-25T11:44:19.106657+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:19.106657+0000","last_undegraded":"2024-04-25T11:44:19.106657+0000","last_fullsized":"2024-04-25T11:44:19.106657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5066","reported_seq":5166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.955500+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:44:59.955500+0000","last_peered":"2024-04-25T11:44:59.955500+0000","last_clean":"2024-04-25T11:44:59.955500+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:44:59.955500+0000","last_undegraded":"2024-04-25T11:44:59.955500+0000","last_fullsized":"2024-04-25T11:44:59.955500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16802835,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21087,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'10","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:52.762774+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:52.762774+0000","last_peered":"2024-04-25T11:44:52.762774+0000","last_clean":"2024-04-25T11:44:52.762774+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:44:52.762774+0000","last_undegraded":"2024-04-25T11:44:52.762774+0000","last_fullsized":"2024-04-25T11:44:52.762774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"37'3","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.119332+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:44:19.119332+0000","last_peered":"2024-04-25T11:44:19.119332+0000","last_clean":"2024-04-25T11:44:19.119332+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:44:19.119332+0000","last_undegraded":"2024-04-25T11:44:19.119332+0000","last_fullsized":"2024-04-25T11:44:19.119332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5290","reported_seq":5396,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.948674+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:45:00.948674+0000","last_peered":"2024-04-25T11:45:00.948674+0000","last_clean":"2024-04-25T11:45:00.948674+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:00.948674+0000","last_undegraded":"2024-04-25T11:45:00.948674+0000","last_fullsized":"2024-04-25T11:45:00.948674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20995437,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21937,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.886490+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:44:45.886490+0000","last_peered":"2024-04-25T11:44:45.886490+0000","last_clean":"2024-04-25T11:44:45.886490+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:44:45.886490+0000","last_undegraded":"2024-04-25T11:44:45.886490+0000","last_fullsized":"2024-04-25T11:44:45.886490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'15","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.175082+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:44:24.175082+0000","last_peered":"2024-04-25T11:44:24.175082+0000","last_clean":"2024-04-25T11:44:24.175082+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:44:24.175082+0000","last_undegraded":"2024-04-25T11:44:24.175082+0000","last_fullsized":"2024-04-25T11:44:24.175082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5436","reported_seq":5546,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.076416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:45:01.076416+0000","last_peered":"2024-04-25T11:45:01.076416+0000","last_clean":"2024-04-25T11:45:01.076416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:01.076416+0000","last_undegraded":"2024-04-25T11:45:01.076416+0000","last_fullsized":"2024-04-25T11:45:01.076416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4210952,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23006,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'9","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111309+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:44:19.111309+0000","last_peered":"2024-04-25T11:44:19.111309+0000","last_clean":"2024-04-25T11:44:19.111309+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:44:19.111309+0000","last_undegraded":"2024-04-25T11:44:19.111309+0000","last_fullsized":"2024-04-25T11:44:19.111309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'64","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482378+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:45:01.482378+0000","last_peered":"2024-04-25T11:45:01.482378+0000","last_clean":"2024-04-25T11:45:01.482378+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:01.482378+0000","last_undegraded":"2024-04-25T11:45:01.482378+0000","last_fullsized":"2024-04-25T11:45:01.482378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5057","reported_seq":5163,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.917230+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:45:00.917230+0000","last_peered":"2024-04-25T11:45:00.917230+0000","last_clean":"2024-04-25T11:45:00.917230+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:00.917230+0000","last_undegraded":"2024-04-25T11:45:00.917230+0000","last_fullsized":"2024-04-25T11:45:00.917230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4277711,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21666,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'5","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.194821+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:44:47.194821+0000","last_peered":"2024-04-25T11:44:47.194821+0000","last_clean":"2024-04-25T11:44:47.194821+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:44:47.194821+0000","last_undegraded":"2024-04-25T11:44:47.194821+0000","last_fullsized":"2024-04-25T11:44:47.194821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.511395+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:44:46.511395+0000","last_peered":"2024-04-25T11:44:46.511395+0000","last_clean":"2024-04-25T11:44:46.511395+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:44:46.511395+0000","last_undegraded":"2024-04-25T11:44:46.511395+0000","last_fullsized":"2024-04-25T11:44:46.511395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5126","reported_seq":5221,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.068297+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:45:00.068297+0000","last_peered":"2024-04-25T11:45:00.068297+0000","last_clean":"2024-04-25T11:45:00.068297+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:00.068297+0000","last_undegraded":"2024-04-25T11:45:00.068297+0000","last_fullsized":"2024-04-25T11:45:00.068297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16791111,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21514,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"26'3","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.129182+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:44:19.129182+0000","last_peered":"2024-04-25T11:44:19.129182+0000","last_clean":"2024-04-25T11:44:19.129182+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:44:19.129182+0000","last_undegraded":"2024-04-25T11:44:19.129182+0000","last_fullsized":"2024-04-25T11:44:19.129182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.087665+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:44:19.087665+0000","last_peered":"2024-04-25T11:44:19.087665+0000","last_clean":"2024-04-25T11:44:19.087665+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:44:19.087665+0000","last_undegraded":"2024-04-25T11:44:19.087665+0000","last_fullsized":"2024-04-25T11:44:19.087665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":162,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105537+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:44:19.105537+0000","last_peered":"2024-04-25T11:44:19.105537+0000","last_clean":"2024-04-25T11:44:19.105537+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:44:19.105537+0000","last_undegraded":"2024-04-25T11:44:19.105537+0000","last_fullsized":"2024-04-25T11:44:19.105537+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5031","reported_seq":5126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.959479+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:45:00.959479+0000","last_peered":"2024-04-25T11:45:00.959479+0000","last_clean":"2024-04-25T11:45:00.959479+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:00.959479+0000","last_undegraded":"2024-04-25T11:45:00.959479+0000","last_fullsized":"2024-04-25T11:45:00.959479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4221465,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21810,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.613620+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:44:59.613620+0000","last_peered":"2024-04-25T11:44:59.613620+0000","last_clean":"2024-04-25T11:44:59.613620+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:44:59.613620+0000","last_undegraded":"2024-04-25T11:44:59.613620+0000","last_fullsized":"2024-04-25T11:44:59.613620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'16","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.725791+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:45:01.725791+0000","last_peered":"2024-04-25T11:45:01.725791+0000","last_clean":"2024-04-25T11:45:01.725791+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:01.725791+0000","last_undegraded":"2024-04-25T11:45:01.725791+0000","last_fullsized":"2024-04-25T11:45:01.725791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5270","reported_seq":5368,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.055983+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:45:01.055983+0000","last_peered":"2024-04-25T11:45:01.055983+0000","last_clean":"2024-04-25T11:45:01.055983+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:01.055983+0000","last_undegraded":"2024-04-25T11:45:01.055983+0000","last_fullsized":"2024-04-25T11:45:01.055983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54882,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22094,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"26'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088836+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:44:19.088836+0000","last_peered":"2024-04-25T11:44:19.088836+0000","last_clean":"2024-04-25T11:44:19.088836+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:44:19.088836+0000","last_undegraded":"2024-04-25T11:44:19.088836+0000","last_fullsized":"2024-04-25T11:44:19.088836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'13","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006942+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:44:56.006942+0000","last_peered":"2024-04-25T11:44:56.006942+0000","last_clean":"2024-04-25T11:44:56.006942+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:44:56.006942+0000","last_undegraded":"2024-04-25T11:44:56.006942+0000","last_fullsized":"2024-04-25T11:44:56.006942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4937","reported_seq":5037,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.189940+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:45:00.189940+0000","last_peered":"2024-04-25T11:45:00.189940+0000","last_clean":"2024-04-25T11:45:00.189940+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:00.189940+0000","last_undegraded":"2024-04-25T11:45:00.189940+0000","last_fullsized":"2024-04-25T11:45:00.189940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311976,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20945,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.239118+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:44:57.239118+0000","last_peered":"2024-04-25T11:44:57.239118+0000","last_clean":"2024-04-25T11:44:57.239118+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:44:57.239118+0000","last_undegraded":"2024-04-25T11:44:57.239118+0000","last_fullsized":"2024-04-25T11:44:57.239118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'12","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.738638+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:45:01.738638+0000","last_peered":"2024-04-25T11:45:01.738638+0000","last_clean":"2024-04-25T11:45:01.738638+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:01.738638+0000","last_undegraded":"2024-04-25T11:45:01.738638+0000","last_fullsized":"2024-04-25T11:45:01.738638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4817","reported_seq":4920,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.962036+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:45:00.962036+0000","last_peered":"2024-04-25T11:45:00.962036+0000","last_clean":"2024-04-25T11:45:00.962036+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:00.962036+0000","last_undegraded":"2024-04-25T11:45:00.962036+0000","last_fullsized":"2024-04-25T11:45:00.962036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18844,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20361,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:44.444922+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:44:44.444922+0000","last_peered":"2024-04-25T11:44:44.444922+0000","last_clean":"2024-04-25T11:44:44.444922+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:44:44.444922+0000","last_undegraded":"2024-04-25T11:44:44.444922+0000","last_fullsized":"2024-04-25T11:44:44.444922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'5","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.084030+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:44:24.084030+0000","last_peered":"2024-04-25T11:44:24.084030+0000","last_clean":"2024-04-25T11:44:24.084030+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:44:24.084030+0000","last_undegraded":"2024-04-25T11:44:24.084030+0000","last_fullsized":"2024-04-25T11:44:24.084030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4347","reported_seq":4442,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.433320+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:45:01.433320+0000","last_peered":"2024-04-25T11:45:01.433320+0000","last_clean":"2024-04-25T11:45:01.433320+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:01.433320+0000","last_undegraded":"2024-04-25T11:45:01.433320+0000","last_fullsized":"2024-04-25T11:45:01.433320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2709,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18463,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.271370+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:44:50.271370+0000","last_peered":"2024-04-25T11:44:50.271370+0000","last_clean":"2024-04-25T11:44:50.271370+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:44:50.271370+0000","last_undegraded":"2024-04-25T11:44:50.271370+0000","last_fullsized":"2024-04-25T11:44:50.271370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'31","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732025+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:45:01.732025+0000","last_peered":"2024-04-25T11:45:01.732025+0000","last_clean":"2024-04-25T11:45:01.732025+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:01.732025+0000","last_undegraded":"2024-04-25T11:45:01.732025+0000","last_fullsized":"2024-04-25T11:45:01.732025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5258","reported_seq":5356,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.801479+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:44:59.801479+0000","last_peered":"2024-04-25T11:44:59.801479+0000","last_clean":"2024-04-25T11:44:59.801479+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:44:59.801479+0000","last_undegraded":"2024-04-25T11:44:59.801479+0000","last_fullsized":"2024-04-25T11:44:59.801479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33583936,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21931,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:43.023104+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:44:43.023104+0000","last_peered":"2024-04-25T11:44:43.023104+0000","last_clean":"2024-04-25T11:44:43.023104+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:44:43.023104+0000","last_undegraded":"2024-04-25T11:44:43.023104+0000","last_fullsized":"2024-04-25T11:44:43.023104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'11","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:40.736132+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:44:40.736132+0000","last_peered":"2024-04-25T11:44:40.736132+0000","last_clean":"2024-04-25T11:44:40.736132+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:44:40.736132+0000","last_undegraded":"2024-04-25T11:44:40.736132+0000","last_fullsized":"2024-04-25T11:44:40.736132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5689","reported_seq":5806,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.213416+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:44:58.213416+0000","last_peered":"2024-04-25T11:44:58.213416+0000","last_clean":"2024-04-25T11:44:58.213416+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:44:58.213416+0000","last_undegraded":"2024-04-25T11:44:58.213416+0000","last_fullsized":"2024-04-25T11:44:58.213416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88343888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23822,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"26'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105615+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:44:19.105615+0000","last_peered":"2024-04-25T11:44:19.105615+0000","last_clean":"2024-04-25T11:44:19.105615+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:44:19.105615+0000","last_undegraded":"2024-04-25T11:44:19.105615+0000","last_fullsized":"2024-04-25T11:44:19.105615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"39'9","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088317+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:44:19.088317+0000","last_peered":"2024-04-25T11:44:19.088317+0000","last_clean":"2024-04-25T11:44:19.088317+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:44:19.088317+0000","last_undegraded":"2024-04-25T11:44:19.088317+0000","last_fullsized":"2024-04-25T11:44:19.088317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4245","reported_seq":4353,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.531793+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:44:58.531793+0000","last_peered":"2024-04-25T11:44:58.531793+0000","last_clean":"2024-04-25T11:44:58.531793+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:44:58.531793+0000","last_undegraded":"2024-04-25T11:44:58.531793+0000","last_fullsized":"2024-04-25T11:44:58.531793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58753411,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17543,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.027133+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:44:55.027133+0000","last_peered":"2024-04-25T11:44:55.027133+0000","last_clean":"2024-04-25T11:44:55.027133+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:44:55.027133+0000","last_undegraded":"2024-04-25T11:44:55.027133+0000","last_fullsized":"2024-04-25T11:44:55.027133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'10","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.512801+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:44:46.512801+0000","last_peered":"2024-04-25T11:44:46.512801+0000","last_clean":"2024-04-25T11:44:46.512801+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:44:46.512801+0000","last_undegraded":"2024-04-25T11:44:46.512801+0000","last_fullsized":"2024-04-25T11:44:46.512801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4278","reported_seq":4372,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.989857+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:45:00.989857+0000","last_peered":"2024-04-25T11:45:00.989857+0000","last_clean":"2024-04-25T11:45:00.989857+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:00.989857+0000","last_undegraded":"2024-04-25T11:45:00.989857+0000","last_fullsized":"2024-04-25T11:45:00.989857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17948,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.022826+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:44:45.022826+0000","last_peered":"2024-04-25T11:44:45.022826+0000","last_clean":"2024-04-25T11:44:45.022826+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:44:45.022826+0000","last_undegraded":"2024-04-25T11:44:45.022826+0000","last_fullsized":"2024-04-25T11:44:45.022826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'348","reported_seq":441,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.649851+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:44:42.649851+0000","last_peered":"2024-04-25T11:44:42.649851+0000","last_clean":"2024-04-25T11:44:42.649851+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:44:42.649851+0000","last_undegraded":"2024-04-25T11:44:42.649851+0000","last_fullsized":"2024-04-25T11:44:42.649851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4214","reported_seq":4312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.997646+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:45:00.997646+0000","last_peered":"2024-04-25T11:45:00.997646+0000","last_clean":"2024-04-25T11:45:00.997646+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:00.997646+0000","last_undegraded":"2024-04-25T11:45:00.997646+0000","last_fullsized":"2024-04-25T11:45:00.997646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240135,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17728,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'8","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.051954+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:44:58.051954+0000","last_peered":"2024-04-25T11:44:58.051954+0000","last_clean":"2024-04-25T11:44:58.051954+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:44:58.051954+0000","last_undegraded":"2024-04-25T11:44:58.051954+0000","last_fullsized":"2024-04-25T11:44:58.051954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'28","reported_seq":103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006617+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:44:56.006617+0000","last_peered":"2024-04-25T11:44:56.006617+0000","last_clean":"2024-04-25T11:44:56.006617+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:44:56.006617+0000","last_undegraded":"2024-04-25T11:44:56.006617+0000","last_fullsized":"2024-04-25T11:44:56.006617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":31,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4469","reported_seq":4558,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.583673+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:45:01.583673+0000","last_peered":"2024-04-25T11:45:01.583673+0000","last_clean":"2024-04-25T11:45:01.583673+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:01.583673+0000","last_undegraded":"2024-04-25T11:45:01.583673+0000","last_fullsized":"2024-04-25T11:45:01.583673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410839,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18507,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.636324+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:44:50.636324+0000","last_peered":"2024-04-25T11:44:50.636324+0000","last_clean":"2024-04-25T11:44:50.636324+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:44:50.636324+0000","last_undegraded":"2024-04-25T11:44:50.636324+0000","last_fullsized":"2024-04-25T11:44:50.636324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.089262+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:44:19.089262+0000","last_peered":"2024-04-25T11:44:19.089262+0000","last_clean":"2024-04-25T11:44:19.089262+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:44:19.089262+0000","last_undegraded":"2024-04-25T11:44:19.089262+0000","last_fullsized":"2024-04-25T11:44:19.089262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4316","reported_seq":4419,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.854217+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:44:58.854217+0000","last_peered":"2024-04-25T11:44:58.854217+0000","last_clean":"2024-04-25T11:44:58.854217+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:44:58.854217+0000","last_undegraded":"2024-04-25T11:44:58.854217+0000","last_fullsized":"2024-04-25T11:44:58.854217+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75500283,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17930,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.055161+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:44:58.055161+0000","last_peered":"2024-04-25T11:44:58.055161+0000","last_clean":"2024-04-25T11:44:58.055161+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:44:58.055161+0000","last_undegraded":"2024-04-25T11:44:58.055161+0000","last_fullsized":"2024-04-25T11:44:58.055161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'14","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.755327+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:44:56.755327+0000","last_peered":"2024-04-25T11:44:56.755327+0000","last_clean":"2024-04-25T11:44:56.755327+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:44:56.755327+0000","last_undegraded":"2024-04-25T11:44:56.755327+0000","last_fullsized":"2024-04-25T11:44:56.755327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3766","reported_seq":3868,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.789112+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:44:58.789112+0000","last_peered":"2024-04-25T11:44:58.789112+0000","last_clean":"2024-04-25T11:44:58.789112+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:44:58.789112+0000","last_undegraded":"2024-04-25T11:44:58.789112+0000","last_fullsized":"2024-04-25T11:44:58.789112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75515216,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15950,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5407","reported_seq":5520,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.592390+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:44:59.592390+0000","last_peered":"2024-04-25T11:44:59.592390+0000","last_clean":"2024-04-25T11:44:59.592390+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:44:59.592390+0000","last_undegraded":"2024-04-25T11:44:59.592390+0000","last_fullsized":"2024-04-25T11:44:59.592390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16805846,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22729,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'4","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.042946+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:44:46.042946+0000","last_peered":"2024-04-25T11:44:46.042946+0000","last_clean":"2024-04-25T11:44:46.042946+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:44:46.042946+0000","last_undegraded":"2024-04-25T11:44:46.042946+0000","last_fullsized":"2024-04-25T11:44:46.042946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'12","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.021180+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:44:31.021180+0000","last_peered":"2024-04-25T11:44:31.021180+0000","last_clean":"2024-04-25T11:44:31.021180+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:44:31.021180+0000","last_undegraded":"2024-04-25T11:44:31.021180+0000","last_fullsized":"2024-04-25T11:44:31.021180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5672","reported_seq":5787,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.033114+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:45:01.033114+0000","last_peered":"2024-04-25T11:45:01.033114+0000","last_clean":"2024-04-25T11:45:01.033114+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:01.033114+0000","last_undegraded":"2024-04-25T11:45:01.033114+0000","last_fullsized":"2024-04-25T11:45:01.033114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8447109,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24242,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:33.648968+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:44:33.648968+0000","last_peered":"2024-04-25T11:44:33.648968+0000","last_clean":"2024-04-25T11:44:33.648968+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:44:33.648968+0000","last_undegraded":"2024-04-25T11:44:33.648968+0000","last_fullsized":"2024-04-25T11:44:33.648968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.507269+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:44:46.507269+0000","last_peered":"2024-04-25T11:44:46.507269+0000","last_clean":"2024-04-25T11:44:46.507269+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:44:46.507269+0000","last_undegraded":"2024-04-25T11:44:46.507269+0000","last_fullsized":"2024-04-25T11:44:46.507269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"38'11","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.510877+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:44:46.510877+0000","last_peered":"2024-04-25T11:44:46.510877+0000","last_clean":"2024-04-25T11:44:46.510877+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:44:46.510877+0000","last_undegraded":"2024-04-25T11:44:46.510877+0000","last_fullsized":"2024-04-25T11:44:46.510877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.709218+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:44:59.709218+0000","last_peered":"2024-04-25T11:44:59.709218+0000","last_clean":"2024-04-25T11:44:59.709218+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:44:59.709218+0000","last_undegraded":"2024-04-25T11:44:59.709218+0000","last_fullsized":"2024-04-25T11:44:59.709218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.368416+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:45:01.368416+0000","last_peered":"2024-04-25T11:45:01.368416+0000","last_clean":"2024-04-25T11:45:01.368416+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:01.368416+0000","last_undegraded":"2024-04-25T11:45:01.368416+0000","last_fullsized":"2024-04-25T11:45:01.368416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'636","reported_seq":710,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.539919+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:44:46.539919+0000","last_peered":"2024-04-25T11:44:46.539919+0000","last_clean":"2024-04-25T11:44:46.539919+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:44:46.539919+0000","last_undegraded":"2024-04-25T11:44:46.539919+0000","last_fullsized":"2024-04-25T11:44:46.539919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.315631+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:44:50.315631+0000","last_peered":"2024-04-25T11:44:50.315631+0000","last_clean":"2024-04-25T11:44:50.315631+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:44:50.315631+0000","last_undegraded":"2024-04-25T11:44:50.315631+0000","last_fullsized":"2024-04-25T11:44:50.315631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5248","reported_seq":5351,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.821173+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:44:58.821173+0000","last_peered":"2024-04-25T11:44:58.821173+0000","last_clean":"2024-04-25T11:44:58.821173+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:44:58.821173+0000","last_undegraded":"2024-04-25T11:44:58.821173+0000","last_fullsized":"2024-04-25T11:44:58.821173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5248,"log_dups_size":0,"ondisk_log_size":5248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46139613,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22077,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.073823+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:44:53.073823+0000","last_peered":"2024-04-25T11:44:53.073823+0000","last_clean":"2024-04-25T11:44:53.073823+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:44:53.073823+0000","last_undegraded":"2024-04-25T11:44:53.073823+0000","last_fullsized":"2024-04-25T11:44:53.073823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.606903+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:45:01.606903+0000","last_peered":"2024-04-25T11:45:01.606903+0000","last_clean":"2024-04-25T11:45:01.606903+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:01.606903+0000","last_undegraded":"2024-04-25T11:45:01.606903+0000","last_fullsized":"2024-04-25T11:45:01.606903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'14","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.513817+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:44:54.513817+0000","last_peered":"2024-04-25T11:44:54.513817+0000","last_clean":"2024-04-25T11:44:54.513817+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:44:54.513817+0000","last_undegraded":"2024-04-25T11:44:54.513817+0000","last_fullsized":"2024-04-25T11:44:54.513817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.120881+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:44:19.120881+0000","last_peered":"2024-04-25T11:44:19.120881+0000","last_clean":"2024-04-25T11:44:19.120881+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:44:19.120881+0000","last_undegraded":"2024-04-25T11:44:19.120881+0000","last_fullsized":"2024-04-25T11:44:19.120881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.338193+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:44:50.338193+0000","last_peered":"2024-04-25T11:44:50.338193+0000","last_clean":"2024-04-25T11:44:50.338193+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:44:50.338193+0000","last_undegraded":"2024-04-25T11:44:50.338193+0000","last_fullsized":"2024-04-25T11:44:50.338193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.012448+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:44:56.012448+0000","last_peered":"2024-04-25T11:44:56.012448+0000","last_clean":"2024-04-25T11:44:56.012448+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:56.012448+0000","last_undegraded":"2024-04-25T11:44:56.012448+0000","last_fullsized":"2024-04-25T11:44:56.012448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.845960+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:44:58.845960+0000","last_peered":"2024-04-25T11:44:58.845960+0000","last_clean":"2024-04-25T11:44:58.845960+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:44:58.845960+0000","last_undegraded":"2024-04-25T11:44:58.845960+0000","last_fullsized":"2024-04-25T11:44:58.845960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.266380+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:44:50.266380+0000","last_peered":"2024-04-25T11:44:50.266380+0000","last_clean":"2024-04-25T11:44:50.266380+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:44:50.266380+0000","last_undegraded":"2024-04-25T11:44:50.266380+0000","last_fullsized":"2024-04-25T11:44:50.266380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.291749+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:44:50.291749+0000","last_peered":"2024-04-25T11:44:50.291749+0000","last_clean":"2024-04-25T11:44:50.291749+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:44:50.291749+0000","last_undegraded":"2024-04-25T11:44:50.291749+0000","last_fullsized":"2024-04-25T11:44:50.291749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.021270+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:44:47.021270+0000","last_peered":"2024-04-25T11:44:47.021270+0000","last_clean":"2024-04-25T11:44:47.021270+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:44:47.021270+0000","last_undegraded":"2024-04-25T11:44:47.021270+0000","last_fullsized":"2024-04-25T11:44:47.021270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.361691+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:44:48.361691+0000","last_peered":"2024-04-25T11:44:48.361691+0000","last_clean":"2024-04-25T11:44:48.361691+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:44:48.361691+0000","last_undegraded":"2024-04-25T11:44:48.361691+0000","last_fullsized":"2024-04-25T11:44:48.361691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.848357+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:44:54.848357+0000","last_peered":"2024-04-25T11:44:54.848357+0000","last_clean":"2024-04-25T11:44:54.848357+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:44:54.848357+0000","last_undegraded":"2024-04-25T11:44:54.848357+0000","last_fullsized":"2024-04-25T11:44:54.848357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.113943+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:44:19.113943+0000","last_peered":"2024-04-25T11:44:19.113943+0000","last_clean":"2024-04-25T11:44:19.113943+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:44:19.113943+0000","last_undegraded":"2024-04-25T11:44:19.113943+0000","last_fullsized":"2024-04-25T11:44:19.113943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'12","reported_seq":60,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:51.941786+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:44:51.941786+0000","last_peered":"2024-04-25T11:44:51.941786+0000","last_clean":"2024-04-25T11:44:51.941786+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:44:51.941786+0000","last_undegraded":"2024-04-25T11:44:51.941786+0000","last_fullsized":"2024-04-25T11:44:51.941786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732164+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:45:01.732164+0000","last_peered":"2024-04-25T11:45:01.732164+0000","last_clean":"2024-04-25T11:45:01.732164+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:01.732164+0000","last_undegraded":"2024-04-25T11:45:01.732164+0000","last_fullsized":"2024-04-25T11:45:01.732164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.825982+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:44:56.825982+0000","last_peered":"2024-04-25T11:44:56.825982+0000","last_clean":"2024-04-25T11:44:56.825982+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:44:56.825982+0000","last_undegraded":"2024-04-25T11:44:56.825982+0000","last_fullsized":"2024-04-25T11:44:56.825982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.844083+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:55.844083+0000","last_peered":"2024-04-25T11:44:55.844083+0000","last_clean":"2024-04-25T11:44:55.844083+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:44:55.844083+0000","last_undegraded":"2024-04-25T11:44:55.844083+0000","last_fullsized":"2024-04-25T11:44:55.844083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.240393+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:44:49.240393+0000","last_peered":"2024-04-25T11:44:49.240393+0000","last_clean":"2024-04-25T11:44:49.240393+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:44:49.240393+0000","last_undegraded":"2024-04-25T11:44:49.240393+0000","last_fullsized":"2024-04-25T11:44:49.240393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.999436+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:44:56.999436+0000","last_peered":"2024-04-25T11:44:56.999436+0000","last_clean":"2024-04-25T11:44:56.999436+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:44:56.999436+0000","last_undegraded":"2024-04-25T11:44:56.999436+0000","last_fullsized":"2024-04-25T11:44:56.999436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'10","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.496713+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:44:50.496713+0000","last_peered":"2024-04-25T11:44:50.496713+0000","last_clean":"2024-04-25T11:44:50.496713+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:44:50.496713+0000","last_undegraded":"2024-04-25T11:44:50.496713+0000","last_fullsized":"2024-04-25T11:44:50.496713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'9","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.323680+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:44:47.323680+0000","last_peered":"2024-04-25T11:44:47.323680+0000","last_clean":"2024-04-25T11:44:47.323680+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:44:47.323680+0000","last_undegraded":"2024-04-25T11:44:47.323680+0000","last_fullsized":"2024-04-25T11:44:47.323680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137154+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:44:31.137154+0000","last_peered":"2024-04-25T11:44:31.137154+0000","last_clean":"2024-04-25T11:44:31.137154+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:44:31.137154+0000","last_undegraded":"2024-04-25T11:44:31.137154+0000","last_fullsized":"2024-04-25T11:44:31.137154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137174+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:44:31.137174+0000","last_peered":"2024-04-25T11:44:31.137174+0000","last_clean":"2024-04-25T11:44:31.137174+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:44:31.137174+0000","last_undegraded":"2024-04-25T11:44:31.137174+0000","last_fullsized":"2024-04-25T11:44:31.137174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'14","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.177913+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:44:56.177913+0000","last_peered":"2024-04-25T11:44:56.177913+0000","last_clean":"2024-04-25T11:44:56.177913+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:44:56.177913+0000","last_undegraded":"2024-04-25T11:44:56.177913+0000","last_fullsized":"2024-04-25T11:44:56.177913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'5","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.663176+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:44:53.663176+0000","last_peered":"2024-04-25T11:44:53.663176+0000","last_clean":"2024-04-25T11:44:53.663176+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:44:53.663176+0000","last_undegraded":"2024-04-25T11:44:53.663176+0000","last_fullsized":"2024-04-25T11:44:53.663176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'6","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.631920+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:44:57.631920+0000","last_peered":"2024-04-25T11:44:57.631920+0000","last_clean":"2024-04-25T11:44:57.631920+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:57.631920+0000","last_undegraded":"2024-04-25T11:44:57.631920+0000","last_fullsized":"2024-04-25T11:44:57.631920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"26'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088628+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:44:19.088628+0000","last_peered":"2024-04-25T11:44:19.088628+0000","last_clean":"2024-04-25T11:44:19.088628+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:44:19.088628+0000","last_undegraded":"2024-04-25T11:44:19.088628+0000","last_fullsized":"2024-04-25T11:44:19.088628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.973048+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:44:56.973048+0000","last_peered":"2024-04-25T11:44:56.973048+0000","last_clean":"2024-04-25T11:44:56.973048+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:44:56.973048+0000","last_undegraded":"2024-04-25T11:44:56.973048+0000","last_fullsized":"2024-04-25T11:44:56.973048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'4","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:26.020661+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:44:26.020661+0000","last_peered":"2024-04-25T11:44:26.020661+0000","last_clean":"2024-04-25T11:44:26.020661+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:44:26.020661+0000","last_undegraded":"2024-04-25T11:44:26.020661+0000","last_fullsized":"2024-04-25T11:44:26.020661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.484060+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:44:49.484060+0000","last_peered":"2024-04-25T11:44:49.484060+0000","last_clean":"2024-04-25T11:44:49.484060+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:44:49.484060+0000","last_undegraded":"2024-04-25T11:44:49.484060+0000","last_fullsized":"2024-04-25T11:44:49.484060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'3","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.122357+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:44:19.122357+0000","last_peered":"2024-04-25T11:44:19.122357+0000","last_clean":"2024-04-25T11:44:19.122357+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:44:19.122357+0000","last_undegraded":"2024-04-25T11:44:19.122357+0000","last_fullsized":"2024-04-25T11:44:19.122357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'5","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108672+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:44:19.108672+0000","last_peered":"2024-04-25T11:44:19.108672+0000","last_clean":"2024-04-25T11:44:19.108672+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:44:19.108672+0000","last_undegraded":"2024-04-25T11:44:19.108672+0000","last_fullsized":"2024-04-25T11:44:19.108672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":626661889,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654654,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1030500352,"data_stored":1030500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156529,"ondisk_log_size":156529,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":592,"ondisk_log_size":592,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3395,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3076,"ondisk_log_size":3076,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129712,"kb_used_data":141352,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93242128,"statfs":{"total":96636764160,"available":95479939072,"internally_reserved":0,"allocated":144744448,"data_stored":144477234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1385500,"kb_used_data":202132,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":92986340,"statfs":{"total":96636764160,"available":95218012160,"internally_reserved":0,"allocated":206983168,"data_stored":206731993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575151,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938684,"kb_used_data":162868,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93433156,"statfs":{"total":96636764160,"available":95675551744,"internally_reserved":0,"allocated":166776832,"data_stored":166503551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1153172,"kb_used_data":25740,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93218668,"statfs":{"total":96636764160,"available":95455916032,"internally_reserved":0,"allocated":26357760,"data_stored":26106453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575147,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2136464,"kb_used_data":715976,"kb_used_omap":1,"kb_used_meta":1420478,"kb_avail":92235376,"statfs":{"total":96636764160,"available":94449025024,"internally_reserved":0,"allocated":733159424,"data_stored":732900448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1454569927},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575149,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":853412,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":852414,"kb_avail":93518428,"statfs":{"total":96636764160,"available":95762870272,"internally_reserved":0,"allocated":1011712,"data_stored":748490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872872389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1570232,"kb_used_data":69104,"kb_used_omap":1,"kb_used_meta":1501118,"kb_avail":92801608,"statfs":{"total":96636764160,"available":95028846592,"internally_reserved":0,"allocated":70762496,"data_stored":70506481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1537145286},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575149,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276228,"kb_used_data":225404,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93095612,"statfs":{"total":96636764160,"available":95329906688,"internally_reserved":0,"allocated":230813696,"data_stored":230552518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183062528,"data_stored":183062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61894656,"data_stored":61894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":648699904,"data_stored":648699904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48836608,"data_stored":48836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42930176,"data_stored":42930176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44556288,"data_stored":44556288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:03.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:03.727 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 Processor -- start 2024-04-25T11:45:03.728 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 -- start start 2024-04-25T11:45:03.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 0x7fc7080cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc708160370 0x7fc708166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc70815c6f0 con 0x7fc70815ff60 2024-04-25T11:45:03.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc7080c6310 con 0x7fc708160370 2024-04-25T11:45:03.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70cd88640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc70815c990 con 0x7fc70815c0d0 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc706d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc708160370 0x7fc708166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 0x7fc7080cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc6fffff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc70815ff60 0x7fc7080ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:50372/0 (socket says 172.21.15.136:50372) 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc6fffff640 1 -- 172.21.15.136:0/1605848093 learned_addr learned my addr 172.21.15.136:0/1605848093 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc706d9d640 1 -- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 msgr2=0x7fc7080cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc706d9d640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 0x7fc7080cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc706d9d640 1 -- 172.21.15.136:0/1605848093 >> v1:172.21.15.157:6789/0 conn(0x7fc70815ff60 legacy=0x7fc7080ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc706d9d640 1 -- 172.21.15.136:0/1605848093 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70815cc30 con 0x7fc708160370 2024-04-25T11:45:03.731 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.743+0000 7fc70659c640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 0x7fc7080cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:03.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc706d9d640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc708160370 0x7fc708166a50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc700009d90 tx=0x7fc70001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f7703957f3ceba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc705d9b640 1 -- 172.21.15.136:0/1605848093 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7000b4070 con 0x7fc708160370 2024-04-25T11:45:03.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc705d9b640 1 -- 172.21.15.136:0/1605848093 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7000af600 con 0x7fc708160370 2024-04-25T11:45:03.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc705d9b640 1 -- 172.21.15.136:0/1605848093 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7000af9d0 con 0x7fc708160370 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 -- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc708160370 msgr2=0x7fc708166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc708160370 0x7fc708166a50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc700009d90 tx=0x7fc70001f150 comp rx=0 tx=0).stop 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 -- 172.21.15.136:0/1605848093 shutdown_connections 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc708160370 0x7fc708166a50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 --2- 172.21.15.136:0/1605848093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc70815c0d0 0x7fc7080cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 -- 172.21.15.136:0/1605848093 >> 172.21.15.136:0/1605848093 conn(0x7fc70806f9d0 msgr2=0x7fc70806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 -- 172.21.15.136:0/1605848093 shutdown_connections 2024-04-25T11:45:03.734 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.747+0000 7fc70cd88640 1 -- 172.21.15.136:0/1605848093 wait complete. 2024-04-25T11:45:03.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 Processor -- start 2024-04-25T11:45:03.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- start start 2024-04-25T11:45:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 0x7fc7081f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc708160370 0x7fc708147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7080cbc60 con 0x7fc70815c0d0 2024-04-25T11:45:03.738 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc70815fa40 con 0x7fc70815ff60 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc70815cf10 con 0x7fc708160370 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44008/0 (socket says 172.21.15.136:44008) 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 -- 172.21.15.136:0/898308372 learned_addr learned my addr 172.21.15.136:0/898308372 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc6fffff640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 0x7fc7081f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc706d9d640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc708160370 0x7fc708147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 -- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc708160370 msgr2=0x7fc708147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc708160370 0x7fc708147a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 -- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 msgr2=0x7fc7081f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 0x7fc7081f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 -- 172.21.15.136:0/898308372 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc708150a70 con 0x7fc70815c0d0 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc6fffff640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 0x7fc7081f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:03.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70659c640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e9b0 tx=0x7fc6f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8017070 con 0x7fc70815c0d0 2024-04-25T11:45:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f800c9a0 con 0x7fc70815c0d0 2024-04-25T11:45:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f800cd70 con 0x7fc70815c0d0 2024-04-25T11:45:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7081f6770 con 0x7fc70815c0d0 2024-04-25T11:45:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.751+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc708155330 con 0x7fc70815c0d0 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70806a350 con 0x7fc70815c0d0 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc6f801a070 con 0x7fc70815c0d0 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc6feffd640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 0x7fc6e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fc6f8090150 con 0x7fc70815c0d0 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc6fffff640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 0x7fc6e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:03.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.755+0000 7fc6fffff640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 0x7fc6e8071910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc70815ab50 tx=0x7fc6f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:03.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.759+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc6f805c860 con 0x7fc70815c0d0 2024-04-25T11:45:03.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.915+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc708159f10 con 0x7fc6e806f400 2024-04-25T11:45:03.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.943+0000 7fc6feffd640 1 -- 172.21.15.136:0/898308372 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334465 (secure 0 0 0) 0x7fc708159f10 con 0x7fc6e806f400 2024-04-25T11:45:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:03.932 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 msgr2=0x7fc6e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 0x7fc6e8071910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc70815ab50 tx=0x7fc6f0009040 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 msgr2=0x7fc708150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800e9b0 tx=0x7fc6f800ee70 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 shutdown_connections 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc6e806f400 0x7fc6e8071910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc708160370 0x7fc708147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc70815ff60 0x7fc7081f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 --2- 172.21.15.136:0/898308372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70815c0d0 0x7fc708150530 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 >> 172.21.15.136:0/898308372 conn(0x7fc70806f9d0 msgr2=0x7fc70815b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 shutdown_connections 2024-04-25T11:45:03.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:03.947+0000 7fc70cd88640 1 -- 172.21.15.136:0/898308372 wait complete. 2024-04-25T11:45:03.951 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-04-25T11:45:03.415956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642572045,"num_objects":385,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658150,"num_write_kb":15736462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160269,"ondisk_log_size":160269,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":10443404,"kb_used_data":1543564,"kb_used_omap":12,"kb_used_meta":8893619,"kb_avail":744531316,"statfs":{"total":773094113280,"available":762400067584,"internally_reserved":0,"allocated":1580609536,"data_stored":1578527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9107066418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,13,8,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8339203214,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2050,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003730"},"pg_stats":[{"pgid":"2.1f","version":"40'272","reported_seq":358,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.557577+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:44:53.557577+0000","last_peered":"2024-04-25T11:44:53.557577+0000","last_clean":"2024-04-25T11:44:53.557577+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:44:53.557577+0000","last_undegraded":"2024-04-25T11:44:53.557577+0000","last_fullsized":"2024-04-25T11:44:53.557577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'4","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588204+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:44:28.588204+0000","last_peered":"2024-04-25T11:44:28.588204+0000","last_clean":"2024-04-25T11:44:28.588204+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:44:28.588204+0000","last_undegraded":"2024-04-25T11:44:28.588204+0000","last_fullsized":"2024-04-25T11:44:28.588204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5319","reported_seq":5424,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.505149+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:44:58.505149+0000","last_peered":"2024-04-25T11:44:58.505149+0000","last_clean":"2024-04-25T11:44:58.505149+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:44:58.505149+0000","last_undegraded":"2024-04-25T11:44:58.505149+0000","last_fullsized":"2024-04-25T11:44:58.505149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67156925,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22037,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'9","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588290+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:44:28.588290+0000","last_peered":"2024-04-25T11:44:28.588290+0000","last_clean":"2024-04-25T11:44:28.588290+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:44:28.588290+0000","last_undegraded":"2024-04-25T11:44:28.588290+0000","last_fullsized":"2024-04-25T11:44:28.588290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.171897+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:44:47.171897+0000","last_peered":"2024-04-25T11:44:47.171897+0000","last_clean":"2024-04-25T11:44:47.171897+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:44:47.171897+0000","last_undegraded":"2024-04-25T11:44:47.171897+0000","last_fullsized":"2024-04-25T11:44:47.171897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4852","reported_seq":4956,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.839347+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:45:00.839347+0000","last_peered":"2024-04-25T11:45:00.839347+0000","last_clean":"2024-04-25T11:45:00.839347+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:00.839347+0000","last_undegraded":"2024-04-25T11:45:00.839347+0000","last_fullsized":"2024-04-25T11:45:00.839347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35243,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20160,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'28","reported_seq":122,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.650684+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:44:42.650684+0000","last_peered":"2024-04-25T11:44:42.650684+0000","last_clean":"2024-04-25T11:44:42.650684+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:42.650684+0000","last_undegraded":"2024-04-25T11:44:42.650684+0000","last_fullsized":"2024-04-25T11:44:42.650684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.299464+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:44:50.299464+0000","last_peered":"2024-04-25T11:44:50.299464+0000","last_clean":"2024-04-25T11:44:50.299464+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:44:50.299464+0000","last_undegraded":"2024-04-25T11:44:50.299464+0000","last_fullsized":"2024-04-25T11:44:50.299464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4262","reported_seq":4357,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.572827+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:45:01.572827+0000","last_peered":"2024-04-25T11:45:01.572827+0000","last_clean":"2024-04-25T11:45:01.572827+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:01.572827+0000","last_undegraded":"2024-04-25T11:45:01.572827+0000","last_fullsized":"2024-04-25T11:45:01.572827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17219,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108293+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:44:19.108293+0000","last_peered":"2024-04-25T11:44:19.108293+0000","last_clean":"2024-04-25T11:44:19.108293+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:44:19.108293+0000","last_undegraded":"2024-04-25T11:44:19.108293+0000","last_fullsized":"2024-04-25T11:44:19.108293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":62,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.649419+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:45:01.649419+0000","last_peered":"2024-04-25T11:45:01.649419+0000","last_clean":"2024-04-25T11:45:01.649419+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:01.649419+0000","last_undegraded":"2024-04-25T11:45:01.649419+0000","last_fullsized":"2024-04-25T11:45:01.649419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4167","reported_seq":4270,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.384290+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:45:01.384290+0000","last_peered":"2024-04-25T11:45:01.384290+0000","last_clean":"2024-04-25T11:45:01.384290+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:01.384290+0000","last_undegraded":"2024-04-25T11:45:01.384290+0000","last_fullsized":"2024-04-25T11:45:01.384290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16706,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"26'6","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.110341+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:44:19.110341+0000","last_peered":"2024-04-25T11:44:19.110341+0000","last_clean":"2024-04-25T11:44:19.110341+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:44:19.110341+0000","last_undegraded":"2024-04-25T11:44:19.110341+0000","last_fullsized":"2024-04-25T11:44:19.110341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1087","reported_seq":1166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105876+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:44:19.105876+0000","last_peered":"2024-04-25T11:44:19.105876+0000","last_clean":"2024-04-25T11:44:19.105876+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:44:19.105876+0000","last_undegraded":"2024-04-25T11:44:19.105876+0000","last_fullsized":"2024-04-25T11:44:19.105876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5157","reported_seq":5255,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482495+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:45:01.482495+0000","last_peered":"2024-04-25T11:45:01.482495+0000","last_clean":"2024-04-25T11:45:01.482495+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:01.482495+0000","last_undegraded":"2024-04-25T11:45:01.482495+0000","last_fullsized":"2024-04-25T11:45:01.482495+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2946,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21466,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"26'5","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.134609+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:44:19.134609+0000","last_peered":"2024-04-25T11:44:19.134609+0000","last_clean":"2024-04-25T11:44:19.134609+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:44:19.134609+0000","last_undegraded":"2024-04-25T11:44:19.134609+0000","last_fullsized":"2024-04-25T11:44:19.134609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"39'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.124512+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:44:19.124512+0000","last_peered":"2024-04-25T11:44:19.124512+0000","last_clean":"2024-04-25T11:44:19.124512+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:44:19.124512+0000","last_undegraded":"2024-04-25T11:44:19.124512+0000","last_fullsized":"2024-04-25T11:44:19.124512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5165","reported_seq":5262,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.463179+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:45:01.463179+0000","last_peered":"2024-04-25T11:45:01.463179+0000","last_clean":"2024-04-25T11:45:01.463179+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:01.463179+0000","last_undegraded":"2024-04-25T11:45:01.463179+0000","last_fullsized":"2024-04-25T11:45:01.463179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15573,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21560,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.325387+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:44:50.325387+0000","last_peered":"2024-04-25T11:44:50.325387+0000","last_clean":"2024-04-25T11:44:50.325387+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:44:50.325387+0000","last_undegraded":"2024-04-25T11:44:50.325387+0000","last_fullsized":"2024-04-25T11:44:50.325387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111004+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:44:19.111004+0000","last_peered":"2024-04-25T11:44:19.111004+0000","last_clean":"2024-04-25T11:44:19.111004+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:44:19.111004+0000","last_undegraded":"2024-04-25T11:44:19.111004+0000","last_fullsized":"2024-04-25T11:44:19.111004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6708,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.767277+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:45:01.767277+0000","last_peered":"2024-04-25T11:45:01.767277+0000","last_clean":"2024-04-25T11:45:01.767277+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:01.767277+0000","last_undegraded":"2024-04-25T11:45:01.767277+0000","last_fullsized":"2024-04-25T11:45:01.767277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.688232+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:44:58.688232+0000","last_peered":"2024-04-25T11:44:58.688232+0000","last_clean":"2024-04-25T11:44:58.688232+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:44:58.688232+0000","last_undegraded":"2024-04-25T11:44:58.688232+0000","last_fullsized":"2024-04-25T11:44:58.688232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'6","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105377+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:44:19.105377+0000","last_peered":"2024-04-25T11:44:19.105377+0000","last_clean":"2024-04-25T11:44:19.105377+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:44:19.105377+0000","last_undegraded":"2024-04-25T11:44:19.105377+0000","last_fullsized":"2024-04-25T11:44:19.105377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4791","reported_seq":4887,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.422519+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:45:01.422519+0000","last_peered":"2024-04-25T11:45:01.422519+0000","last_clean":"2024-04-25T11:45:01.422519+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:01.422519+0000","last_undegraded":"2024-04-25T11:45:01.422519+0000","last_fullsized":"2024-04-25T11:45:01.422519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19671,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'17","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.332441+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:44:50.332441+0000","last_peered":"2024-04-25T11:44:50.332441+0000","last_clean":"2024-04-25T11:44:50.332441+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:44:50.332441+0000","last_undegraded":"2024-04-25T11:44:50.332441+0000","last_fullsized":"2024-04-25T11:44:50.332441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'10","reported_seq":97,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111925+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:44:19.111925+0000","last_peered":"2024-04-25T11:44:19.111925+0000","last_clean":"2024-04-25T11:44:19.111925+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:44:19.111925+0000","last_undegraded":"2024-04-25T11:44:19.111925+0000","last_fullsized":"2024-04-25T11:44:19.111925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4481","reported_seq":4582,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.591684+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:45:01.591684+0000","last_peered":"2024-04-25T11:45:01.591684+0000","last_clean":"2024-04-25T11:45:01.591684+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:01.591684+0000","last_undegraded":"2024-04-25T11:45:01.591684+0000","last_fullsized":"2024-04-25T11:45:01.591684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18512,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"26'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.174729+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:44:19.174729+0000","last_peered":"2024-04-25T11:44:19.174729+0000","last_clean":"2024-04-25T11:44:19.174729+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:44:19.174729+0000","last_undegraded":"2024-04-25T11:44:19.174729+0000","last_fullsized":"2024-04-25T11:44:19.174729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'367","reported_seq":447,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.589654+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:44:48.589654+0000","last_peered":"2024-04-25T11:44:48.589654+0000","last_clean":"2024-04-25T11:44:48.589654+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:44:48.589654+0000","last_undegraded":"2024-04-25T11:44:48.589654+0000","last_fullsized":"2024-04-25T11:44:48.589654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4523","reported_seq":4623,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.545359+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:44:58.545359+0000","last_peered":"2024-04-25T11:44:58.545359+0000","last_clean":"2024-04-25T11:44:58.545359+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:44:58.545359+0000","last_undegraded":"2024-04-25T11:44:58.545359+0000","last_fullsized":"2024-04-25T11:44:58.545359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58795404,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19016,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'7","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.341334+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:44:47.341334+0000","last_peered":"2024-04-25T11:44:47.341334+0000","last_clean":"2024-04-25T11:44:47.341334+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:44:47.341334+0000","last_undegraded":"2024-04-25T11:44:47.341334+0000","last_fullsized":"2024-04-25T11:44:47.341334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'6","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.106657+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:44:19.106657+0000","last_peered":"2024-04-25T11:44:19.106657+0000","last_clean":"2024-04-25T11:44:19.106657+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:19.106657+0000","last_undegraded":"2024-04-25T11:44:19.106657+0000","last_fullsized":"2024-04-25T11:44:19.106657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5066","reported_seq":5166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.955500+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:44:59.955500+0000","last_peered":"2024-04-25T11:44:59.955500+0000","last_clean":"2024-04-25T11:44:59.955500+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:44:59.955500+0000","last_undegraded":"2024-04-25T11:44:59.955500+0000","last_fullsized":"2024-04-25T11:44:59.955500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16802835,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21087,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'10","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:52.762774+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:52.762774+0000","last_peered":"2024-04-25T11:44:52.762774+0000","last_clean":"2024-04-25T11:44:52.762774+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:44:52.762774+0000","last_undegraded":"2024-04-25T11:44:52.762774+0000","last_fullsized":"2024-04-25T11:44:52.762774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"37'3","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.119332+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:44:19.119332+0000","last_peered":"2024-04-25T11:44:19.119332+0000","last_clean":"2024-04-25T11:44:19.119332+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:44:19.119332+0000","last_undegraded":"2024-04-25T11:44:19.119332+0000","last_fullsized":"2024-04-25T11:44:19.119332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5290","reported_seq":5396,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.948674+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:45:00.948674+0000","last_peered":"2024-04-25T11:45:00.948674+0000","last_clean":"2024-04-25T11:45:00.948674+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:00.948674+0000","last_undegraded":"2024-04-25T11:45:00.948674+0000","last_fullsized":"2024-04-25T11:45:00.948674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20995437,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21937,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.886490+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:44:45.886490+0000","last_peered":"2024-04-25T11:44:45.886490+0000","last_clean":"2024-04-25T11:44:45.886490+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:44:45.886490+0000","last_undegraded":"2024-04-25T11:44:45.886490+0000","last_fullsized":"2024-04-25T11:44:45.886490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'15","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.175082+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:44:24.175082+0000","last_peered":"2024-04-25T11:44:24.175082+0000","last_clean":"2024-04-25T11:44:24.175082+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:44:24.175082+0000","last_undegraded":"2024-04-25T11:44:24.175082+0000","last_fullsized":"2024-04-25T11:44:24.175082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5436","reported_seq":5546,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.076416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:45:01.076416+0000","last_peered":"2024-04-25T11:45:01.076416+0000","last_clean":"2024-04-25T11:45:01.076416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:01.076416+0000","last_undegraded":"2024-04-25T11:45:01.076416+0000","last_fullsized":"2024-04-25T11:45:01.076416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4210952,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23006,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'9","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111309+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:44:19.111309+0000","last_peered":"2024-04-25T11:44:19.111309+0000","last_clean":"2024-04-25T11:44:19.111309+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:44:19.111309+0000","last_undegraded":"2024-04-25T11:44:19.111309+0000","last_fullsized":"2024-04-25T11:44:19.111309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'64","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482378+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:45:01.482378+0000","last_peered":"2024-04-25T11:45:01.482378+0000","last_clean":"2024-04-25T11:45:01.482378+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:01.482378+0000","last_undegraded":"2024-04-25T11:45:01.482378+0000","last_fullsized":"2024-04-25T11:45:01.482378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5057","reported_seq":5163,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.917230+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:45:00.917230+0000","last_peered":"2024-04-25T11:45:00.917230+0000","last_clean":"2024-04-25T11:45:00.917230+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:00.917230+0000","last_undegraded":"2024-04-25T11:45:00.917230+0000","last_fullsized":"2024-04-25T11:45:00.917230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4277711,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21666,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'5","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.194821+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:44:47.194821+0000","last_peered":"2024-04-25T11:44:47.194821+0000","last_clean":"2024-04-25T11:44:47.194821+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:44:47.194821+0000","last_undegraded":"2024-04-25T11:44:47.194821+0000","last_fullsized":"2024-04-25T11:44:47.194821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.511395+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:44:46.511395+0000","last_peered":"2024-04-25T11:44:46.511395+0000","last_clean":"2024-04-25T11:44:46.511395+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:44:46.511395+0000","last_undegraded":"2024-04-25T11:44:46.511395+0000","last_fullsized":"2024-04-25T11:44:46.511395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5126","reported_seq":5221,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.068297+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:45:00.068297+0000","last_peered":"2024-04-25T11:45:00.068297+0000","last_clean":"2024-04-25T11:45:00.068297+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:00.068297+0000","last_undegraded":"2024-04-25T11:45:00.068297+0000","last_fullsized":"2024-04-25T11:45:00.068297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16791111,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21514,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"26'3","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.129182+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:44:19.129182+0000","last_peered":"2024-04-25T11:44:19.129182+0000","last_clean":"2024-04-25T11:44:19.129182+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:44:19.129182+0000","last_undegraded":"2024-04-25T11:44:19.129182+0000","last_fullsized":"2024-04-25T11:44:19.129182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.087665+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:44:19.087665+0000","last_peered":"2024-04-25T11:44:19.087665+0000","last_clean":"2024-04-25T11:44:19.087665+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:44:19.087665+0000","last_undegraded":"2024-04-25T11:44:19.087665+0000","last_fullsized":"2024-04-25T11:44:19.087665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":162,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105537+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:44:19.105537+0000","last_peered":"2024-04-25T11:44:19.105537+0000","last_clean":"2024-04-25T11:44:19.105537+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:44:19.105537+0000","last_undegraded":"2024-04-25T11:44:19.105537+0000","last_fullsized":"2024-04-25T11:44:19.105537+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5031","reported_seq":5126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.959479+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:45:00.959479+0000","last_peered":"2024-04-25T11:45:00.959479+0000","last_clean":"2024-04-25T11:45:00.959479+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:00.959479+0000","last_undegraded":"2024-04-25T11:45:00.959479+0000","last_fullsized":"2024-04-25T11:45:00.959479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4221465,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21810,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.613620+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:44:59.613620+0000","last_peered":"2024-04-25T11:44:59.613620+0000","last_clean":"2024-04-25T11:44:59.613620+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:44:59.613620+0000","last_undegraded":"2024-04-25T11:44:59.613620+0000","last_fullsized":"2024-04-25T11:44:59.613620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'16","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.725791+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:45:01.725791+0000","last_peered":"2024-04-25T11:45:01.725791+0000","last_clean":"2024-04-25T11:45:01.725791+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:01.725791+0000","last_undegraded":"2024-04-25T11:45:01.725791+0000","last_fullsized":"2024-04-25T11:45:01.725791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5270","reported_seq":5368,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.055983+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:45:01.055983+0000","last_peered":"2024-04-25T11:45:01.055983+0000","last_clean":"2024-04-25T11:45:01.055983+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:01.055983+0000","last_undegraded":"2024-04-25T11:45:01.055983+0000","last_fullsized":"2024-04-25T11:45:01.055983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54882,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22094,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"26'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088836+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:44:19.088836+0000","last_peered":"2024-04-25T11:44:19.088836+0000","last_clean":"2024-04-25T11:44:19.088836+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:44:19.088836+0000","last_undegraded":"2024-04-25T11:44:19.088836+0000","last_fullsized":"2024-04-25T11:44:19.088836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'13","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006942+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:44:56.006942+0000","last_peered":"2024-04-25T11:44:56.006942+0000","last_clean":"2024-04-25T11:44:56.006942+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:44:56.006942+0000","last_undegraded":"2024-04-25T11:44:56.006942+0000","last_fullsized":"2024-04-25T11:44:56.006942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4937","reported_seq":5037,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.189940+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:45:00.189940+0000","last_peered":"2024-04-25T11:45:00.189940+0000","last_clean":"2024-04-25T11:45:00.189940+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:00.189940+0000","last_undegraded":"2024-04-25T11:45:00.189940+0000","last_fullsized":"2024-04-25T11:45:00.189940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311976,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20945,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.239118+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:44:57.239118+0000","last_peered":"2024-04-25T11:44:57.239118+0000","last_clean":"2024-04-25T11:44:57.239118+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:44:57.239118+0000","last_undegraded":"2024-04-25T11:44:57.239118+0000","last_fullsized":"2024-04-25T11:44:57.239118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'12","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.738638+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:45:01.738638+0000","last_peered":"2024-04-25T11:45:01.738638+0000","last_clean":"2024-04-25T11:45:01.738638+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:01.738638+0000","last_undegraded":"2024-04-25T11:45:01.738638+0000","last_fullsized":"2024-04-25T11:45:01.738638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4817","reported_seq":4920,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.962036+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:45:00.962036+0000","last_peered":"2024-04-25T11:45:00.962036+0000","last_clean":"2024-04-25T11:45:00.962036+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:00.962036+0000","last_undegraded":"2024-04-25T11:45:00.962036+0000","last_fullsized":"2024-04-25T11:45:00.962036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18844,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20361,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:44.444922+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:44:44.444922+0000","last_peered":"2024-04-25T11:44:44.444922+0000","last_clean":"2024-04-25T11:44:44.444922+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:44:44.444922+0000","last_undegraded":"2024-04-25T11:44:44.444922+0000","last_fullsized":"2024-04-25T11:44:44.444922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'5","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.084030+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:44:24.084030+0000","last_peered":"2024-04-25T11:44:24.084030+0000","last_clean":"2024-04-25T11:44:24.084030+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:44:24.084030+0000","last_undegraded":"2024-04-25T11:44:24.084030+0000","last_fullsized":"2024-04-25T11:44:24.084030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4347","reported_seq":4442,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.433320+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:45:01.433320+0000","last_peered":"2024-04-25T11:45:01.433320+0000","last_clean":"2024-04-25T11:45:01.433320+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:01.433320+0000","last_undegraded":"2024-04-25T11:45:01.433320+0000","last_fullsized":"2024-04-25T11:45:01.433320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2709,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18463,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.271370+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:44:50.271370+0000","last_peered":"2024-04-25T11:44:50.271370+0000","last_clean":"2024-04-25T11:44:50.271370+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:44:50.271370+0000","last_undegraded":"2024-04-25T11:44:50.271370+0000","last_fullsized":"2024-04-25T11:44:50.271370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'31","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732025+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:45:01.732025+0000","last_peered":"2024-04-25T11:45:01.732025+0000","last_clean":"2024-04-25T11:45:01.732025+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:01.732025+0000","last_undegraded":"2024-04-25T11:45:01.732025+0000","last_fullsized":"2024-04-25T11:45:01.732025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5258","reported_seq":5356,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.801479+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:44:59.801479+0000","last_peered":"2024-04-25T11:44:59.801479+0000","last_clean":"2024-04-25T11:44:59.801479+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:44:59.801479+0000","last_undegraded":"2024-04-25T11:44:59.801479+0000","last_fullsized":"2024-04-25T11:44:59.801479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33583936,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21931,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:43.023104+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:44:43.023104+0000","last_peered":"2024-04-25T11:44:43.023104+0000","last_clean":"2024-04-25T11:44:43.023104+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:44:43.023104+0000","last_undegraded":"2024-04-25T11:44:43.023104+0000","last_fullsized":"2024-04-25T11:44:43.023104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'11","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:40.736132+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:44:40.736132+0000","last_peered":"2024-04-25T11:44:40.736132+0000","last_clean":"2024-04-25T11:44:40.736132+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:44:40.736132+0000","last_undegraded":"2024-04-25T11:44:40.736132+0000","last_fullsized":"2024-04-25T11:44:40.736132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5689","reported_seq":5806,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.213416+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:44:58.213416+0000","last_peered":"2024-04-25T11:44:58.213416+0000","last_clean":"2024-04-25T11:44:58.213416+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:44:58.213416+0000","last_undegraded":"2024-04-25T11:44:58.213416+0000","last_fullsized":"2024-04-25T11:44:58.213416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88343888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23822,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"26'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105615+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:44:19.105615+0000","last_peered":"2024-04-25T11:44:19.105615+0000","last_clean":"2024-04-25T11:44:19.105615+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:44:19.105615+0000","last_undegraded":"2024-04-25T11:44:19.105615+0000","last_fullsized":"2024-04-25T11:44:19.105615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"39'9","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088317+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:44:19.088317+0000","last_peered":"2024-04-25T11:44:19.088317+0000","last_clean":"2024-04-25T11:44:19.088317+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:44:19.088317+0000","last_undegraded":"2024-04-25T11:44:19.088317+0000","last_fullsized":"2024-04-25T11:44:19.088317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4245","reported_seq":4353,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.531793+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:44:58.531793+0000","last_peered":"2024-04-25T11:44:58.531793+0000","last_clean":"2024-04-25T11:44:58.531793+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:44:58.531793+0000","last_undegraded":"2024-04-25T11:44:58.531793+0000","last_fullsized":"2024-04-25T11:44:58.531793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58753411,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17543,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.027133+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:44:55.027133+0000","last_peered":"2024-04-25T11:44:55.027133+0000","last_clean":"2024-04-25T11:44:55.027133+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:44:55.027133+0000","last_undegraded":"2024-04-25T11:44:55.027133+0000","last_fullsized":"2024-04-25T11:44:55.027133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'10","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.512801+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:44:46.512801+0000","last_peered":"2024-04-25T11:44:46.512801+0000","last_clean":"2024-04-25T11:44:46.512801+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:44:46.512801+0000","last_undegraded":"2024-04-25T11:44:46.512801+0000","last_fullsized":"2024-04-25T11:44:46.512801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4278","reported_seq":4372,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.989857+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:45:00.989857+0000","last_peered":"2024-04-25T11:45:00.989857+0000","last_clean":"2024-04-25T11:45:00.989857+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:00.989857+0000","last_undegraded":"2024-04-25T11:45:00.989857+0000","last_fullsized":"2024-04-25T11:45:00.989857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17948,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.022826+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:44:45.022826+0000","last_peered":"2024-04-25T11:44:45.022826+0000","last_clean":"2024-04-25T11:44:45.022826+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:44:45.022826+0000","last_undegraded":"2024-04-25T11:44:45.022826+0000","last_fullsized":"2024-04-25T11:44:45.022826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'348","reported_seq":441,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.649851+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:44:42.649851+0000","last_peered":"2024-04-25T11:44:42.649851+0000","last_clean":"2024-04-25T11:44:42.649851+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:44:42.649851+0000","last_undegraded":"2024-04-25T11:44:42.649851+0000","last_fullsized":"2024-04-25T11:44:42.649851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4214","reported_seq":4312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.997646+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:45:00.997646+0000","last_peered":"2024-04-25T11:45:00.997646+0000","last_clean":"2024-04-25T11:45:00.997646+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:00.997646+0000","last_undegraded":"2024-04-25T11:45:00.997646+0000","last_fullsized":"2024-04-25T11:45:00.997646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240135,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17728,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'8","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.051954+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:44:58.051954+0000","last_peered":"2024-04-25T11:44:58.051954+0000","last_clean":"2024-04-25T11:44:58.051954+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:44:58.051954+0000","last_undegraded":"2024-04-25T11:44:58.051954+0000","last_fullsized":"2024-04-25T11:44:58.051954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'28","reported_seq":103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006617+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:44:56.006617+0000","last_peered":"2024-04-25T11:44:56.006617+0000","last_clean":"2024-04-25T11:44:56.006617+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:44:56.006617+0000","last_undegraded":"2024-04-25T11:44:56.006617+0000","last_fullsized":"2024-04-25T11:44:56.006617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":31,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4469","reported_seq":4558,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.583673+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:45:01.583673+0000","last_peered":"2024-04-25T11:45:01.583673+0000","last_clean":"2024-04-25T11:45:01.583673+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:01.583673+0000","last_undegraded":"2024-04-25T11:45:01.583673+0000","last_fullsized":"2024-04-25T11:45:01.583673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410839,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18507,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.636324+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:44:50.636324+0000","last_peered":"2024-04-25T11:44:50.636324+0000","last_clean":"2024-04-25T11:44:50.636324+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:44:50.636324+0000","last_undegraded":"2024-04-25T11:44:50.636324+0000","last_fullsized":"2024-04-25T11:44:50.636324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.089262+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:44:19.089262+0000","last_peered":"2024-04-25T11:44:19.089262+0000","last_clean":"2024-04-25T11:44:19.089262+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:44:19.089262+0000","last_undegraded":"2024-04-25T11:44:19.089262+0000","last_fullsized":"2024-04-25T11:44:19.089262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4316","reported_seq":4419,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.854217+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:44:58.854217+0000","last_peered":"2024-04-25T11:44:58.854217+0000","last_clean":"2024-04-25T11:44:58.854217+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:44:58.854217+0000","last_undegraded":"2024-04-25T11:44:58.854217+0000","last_fullsized":"2024-04-25T11:44:58.854217+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75500283,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17930,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.055161+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:44:58.055161+0000","last_peered":"2024-04-25T11:44:58.055161+0000","last_clean":"2024-04-25T11:44:58.055161+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:44:58.055161+0000","last_undegraded":"2024-04-25T11:44:58.055161+0000","last_fullsized":"2024-04-25T11:44:58.055161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'14","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.755327+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:44:56.755327+0000","last_peered":"2024-04-25T11:44:56.755327+0000","last_clean":"2024-04-25T11:44:56.755327+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:44:56.755327+0000","last_undegraded":"2024-04-25T11:44:56.755327+0000","last_fullsized":"2024-04-25T11:44:56.755327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3766","reported_seq":3868,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.789112+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:44:58.789112+0000","last_peered":"2024-04-25T11:44:58.789112+0000","last_clean":"2024-04-25T11:44:58.789112+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:44:58.789112+0000","last_undegraded":"2024-04-25T11:44:58.789112+0000","last_fullsized":"2024-04-25T11:44:58.789112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75515216,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15950,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5407","reported_seq":5520,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.592390+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:44:59.592390+0000","last_peered":"2024-04-25T11:44:59.592390+0000","last_clean":"2024-04-25T11:44:59.592390+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:44:59.592390+0000","last_undegraded":"2024-04-25T11:44:59.592390+0000","last_fullsized":"2024-04-25T11:44:59.592390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16805846,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22729,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'4","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.042946+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:44:46.042946+0000","last_peered":"2024-04-25T11:44:46.042946+0000","last_clean":"2024-04-25T11:44:46.042946+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:44:46.042946+0000","last_undegraded":"2024-04-25T11:44:46.042946+0000","last_fullsized":"2024-04-25T11:44:46.042946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'12","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.021180+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:44:31.021180+0000","last_peered":"2024-04-25T11:44:31.021180+0000","last_clean":"2024-04-25T11:44:31.021180+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:44:31.021180+0000","last_undegraded":"2024-04-25T11:44:31.021180+0000","last_fullsized":"2024-04-25T11:44:31.021180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5672","reported_seq":5787,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.033114+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:45:01.033114+0000","last_peered":"2024-04-25T11:45:01.033114+0000","last_clean":"2024-04-25T11:45:01.033114+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:01.033114+0000","last_undegraded":"2024-04-25T11:45:01.033114+0000","last_fullsized":"2024-04-25T11:45:01.033114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8447109,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24242,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:33.648968+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:44:33.648968+0000","last_peered":"2024-04-25T11:44:33.648968+0000","last_clean":"2024-04-25T11:44:33.648968+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:44:33.648968+0000","last_undegraded":"2024-04-25T11:44:33.648968+0000","last_fullsized":"2024-04-25T11:44:33.648968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.507269+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:44:46.507269+0000","last_peered":"2024-04-25T11:44:46.507269+0000","last_clean":"2024-04-25T11:44:46.507269+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:44:46.507269+0000","last_undegraded":"2024-04-25T11:44:46.507269+0000","last_fullsized":"2024-04-25T11:44:46.507269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"38'11","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.510877+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:44:46.510877+0000","last_peered":"2024-04-25T11:44:46.510877+0000","last_clean":"2024-04-25T11:44:46.510877+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:44:46.510877+0000","last_undegraded":"2024-04-25T11:44:46.510877+0000","last_fullsized":"2024-04-25T11:44:46.510877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.709218+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:44:59.709218+0000","last_peered":"2024-04-25T11:44:59.709218+0000","last_clean":"2024-04-25T11:44:59.709218+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:44:59.709218+0000","last_undegraded":"2024-04-25T11:44:59.709218+0000","last_fullsized":"2024-04-25T11:44:59.709218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.368416+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:45:01.368416+0000","last_peered":"2024-04-25T11:45:01.368416+0000","last_clean":"2024-04-25T11:45:01.368416+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:01.368416+0000","last_undegraded":"2024-04-25T11:45:01.368416+0000","last_fullsized":"2024-04-25T11:45:01.368416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'636","reported_seq":710,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.539919+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:44:46.539919+0000","last_peered":"2024-04-25T11:44:46.539919+0000","last_clean":"2024-04-25T11:44:46.539919+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:44:46.539919+0000","last_undegraded":"2024-04-25T11:44:46.539919+0000","last_fullsized":"2024-04-25T11:44:46.539919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.315631+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:44:50.315631+0000","last_peered":"2024-04-25T11:44:50.315631+0000","last_clean":"2024-04-25T11:44:50.315631+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:44:50.315631+0000","last_undegraded":"2024-04-25T11:44:50.315631+0000","last_fullsized":"2024-04-25T11:44:50.315631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5248","reported_seq":5351,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.821173+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:44:58.821173+0000","last_peered":"2024-04-25T11:44:58.821173+0000","last_clean":"2024-04-25T11:44:58.821173+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:44:58.821173+0000","last_undegraded":"2024-04-25T11:44:58.821173+0000","last_fullsized":"2024-04-25T11:44:58.821173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5248,"log_dups_size":0,"ondisk_log_size":5248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46139613,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22077,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.073823+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:44:53.073823+0000","last_peered":"2024-04-25T11:44:53.073823+0000","last_clean":"2024-04-25T11:44:53.073823+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:44:53.073823+0000","last_undegraded":"2024-04-25T11:44:53.073823+0000","last_fullsized":"2024-04-25T11:44:53.073823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.606903+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:45:01.606903+0000","last_peered":"2024-04-25T11:45:01.606903+0000","last_clean":"2024-04-25T11:45:01.606903+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:01.606903+0000","last_undegraded":"2024-04-25T11:45:01.606903+0000","last_fullsized":"2024-04-25T11:45:01.606903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'14","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.513817+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:44:54.513817+0000","last_peered":"2024-04-25T11:44:54.513817+0000","last_clean":"2024-04-25T11:44:54.513817+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:44:54.513817+0000","last_undegraded":"2024-04-25T11:44:54.513817+0000","last_fullsized":"2024-04-25T11:44:54.513817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.120881+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:44:19.120881+0000","last_peered":"2024-04-25T11:44:19.120881+0000","last_clean":"2024-04-25T11:44:19.120881+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:44:19.120881+0000","last_undegraded":"2024-04-25T11:44:19.120881+0000","last_fullsized":"2024-04-25T11:44:19.120881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.338193+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:44:50.338193+0000","last_peered":"2024-04-25T11:44:50.338193+0000","last_clean":"2024-04-25T11:44:50.338193+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:44:50.338193+0000","last_undegraded":"2024-04-25T11:44:50.338193+0000","last_fullsized":"2024-04-25T11:44:50.338193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.012448+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:44:56.012448+0000","last_peered":"2024-04-25T11:44:56.012448+0000","last_clean":"2024-04-25T11:44:56.012448+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:56.012448+0000","last_undegraded":"2024-04-25T11:44:56.012448+0000","last_fullsized":"2024-04-25T11:44:56.012448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.845960+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:44:58.845960+0000","last_peered":"2024-04-25T11:44:58.845960+0000","last_clean":"2024-04-25T11:44:58.845960+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:44:58.845960+0000","last_undegraded":"2024-04-25T11:44:58.845960+0000","last_fullsized":"2024-04-25T11:44:58.845960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.266380+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:44:50.266380+0000","last_peered":"2024-04-25T11:44:50.266380+0000","last_clean":"2024-04-25T11:44:50.266380+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:44:50.266380+0000","last_undegraded":"2024-04-25T11:44:50.266380+0000","last_fullsized":"2024-04-25T11:44:50.266380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.291749+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:44:50.291749+0000","last_peered":"2024-04-25T11:44:50.291749+0000","last_clean":"2024-04-25T11:44:50.291749+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:44:50.291749+0000","last_undegraded":"2024-04-25T11:44:50.291749+0000","last_fullsized":"2024-04-25T11:44:50.291749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.021270+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:44:47.021270+0000","last_peered":"2024-04-25T11:44:47.021270+0000","last_clean":"2024-04-25T11:44:47.021270+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:44:47.021270+0000","last_undegraded":"2024-04-25T11:44:47.021270+0000","last_fullsized":"2024-04-25T11:44:47.021270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.361691+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:44:48.361691+0000","last_peered":"2024-04-25T11:44:48.361691+0000","last_clean":"2024-04-25T11:44:48.361691+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:44:48.361691+0000","last_undegraded":"2024-04-25T11:44:48.361691+0000","last_fullsized":"2024-04-25T11:44:48.361691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.848357+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:44:54.848357+0000","last_peered":"2024-04-25T11:44:54.848357+0000","last_clean":"2024-04-25T11:44:54.848357+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:44:54.848357+0000","last_undegraded":"2024-04-25T11:44:54.848357+0000","last_fullsized":"2024-04-25T11:44:54.848357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.113943+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:44:19.113943+0000","last_peered":"2024-04-25T11:44:19.113943+0000","last_clean":"2024-04-25T11:44:19.113943+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:44:19.113943+0000","last_undegraded":"2024-04-25T11:44:19.113943+0000","last_fullsized":"2024-04-25T11:44:19.113943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'12","reported_seq":60,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:51.941786+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:44:51.941786+0000","last_peered":"2024-04-25T11:44:51.941786+0000","last_clean":"2024-04-25T11:44:51.941786+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:44:51.941786+0000","last_undegraded":"2024-04-25T11:44:51.941786+0000","last_fullsized":"2024-04-25T11:44:51.941786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732164+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:45:01.732164+0000","last_peered":"2024-04-25T11:45:01.732164+0000","last_clean":"2024-04-25T11:45:01.732164+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:01.732164+0000","last_undegraded":"2024-04-25T11:45:01.732164+0000","last_fullsized":"2024-04-25T11:45:01.732164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.825982+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:44:56.825982+0000","last_peered":"2024-04-25T11:44:56.825982+0000","last_clean":"2024-04-25T11:44:56.825982+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:44:56.825982+0000","last_undegraded":"2024-04-25T11:44:56.825982+0000","last_fullsized":"2024-04-25T11:44:56.825982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.844083+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:55.844083+0000","last_peered":"2024-04-25T11:44:55.844083+0000","last_clean":"2024-04-25T11:44:55.844083+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:44:55.844083+0000","last_undegraded":"2024-04-25T11:44:55.844083+0000","last_fullsized":"2024-04-25T11:44:55.844083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.240393+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:44:49.240393+0000","last_peered":"2024-04-25T11:44:49.240393+0000","last_clean":"2024-04-25T11:44:49.240393+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:44:49.240393+0000","last_undegraded":"2024-04-25T11:44:49.240393+0000","last_fullsized":"2024-04-25T11:44:49.240393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.999436+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:44:56.999436+0000","last_peered":"2024-04-25T11:44:56.999436+0000","last_clean":"2024-04-25T11:44:56.999436+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:44:56.999436+0000","last_undegraded":"2024-04-25T11:44:56.999436+0000","last_fullsized":"2024-04-25T11:44:56.999436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'10","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.496713+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:44:50.496713+0000","last_peered":"2024-04-25T11:44:50.496713+0000","last_clean":"2024-04-25T11:44:50.496713+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:44:50.496713+0000","last_undegraded":"2024-04-25T11:44:50.496713+0000","last_fullsized":"2024-04-25T11:44:50.496713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'9","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.323680+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:44:47.323680+0000","last_peered":"2024-04-25T11:44:47.323680+0000","last_clean":"2024-04-25T11:44:47.323680+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:44:47.323680+0000","last_undegraded":"2024-04-25T11:44:47.323680+0000","last_fullsized":"2024-04-25T11:44:47.323680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137154+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:44:31.137154+0000","last_peered":"2024-04-25T11:44:31.137154+0000","last_clean":"2024-04-25T11:44:31.137154+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:44:31.137154+0000","last_undegraded":"2024-04-25T11:44:31.137154+0000","last_fullsized":"2024-04-25T11:44:31.137154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137174+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:44:31.137174+0000","last_peered":"2024-04-25T11:44:31.137174+0000","last_clean":"2024-04-25T11:44:31.137174+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:44:31.137174+0000","last_undegraded":"2024-04-25T11:44:31.137174+0000","last_fullsized":"2024-04-25T11:44:31.137174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'14","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.177913+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:44:56.177913+0000","last_peered":"2024-04-25T11:44:56.177913+0000","last_clean":"2024-04-25T11:44:56.177913+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:44:56.177913+0000","last_undegraded":"2024-04-25T11:44:56.177913+0000","last_fullsized":"2024-04-25T11:44:56.177913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'5","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.663176+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:44:53.663176+0000","last_peered":"2024-04-25T11:44:53.663176+0000","last_clean":"2024-04-25T11:44:53.663176+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:44:53.663176+0000","last_undegraded":"2024-04-25T11:44:53.663176+0000","last_fullsized":"2024-04-25T11:44:53.663176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'6","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.631920+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:44:57.631920+0000","last_peered":"2024-04-25T11:44:57.631920+0000","last_clean":"2024-04-25T11:44:57.631920+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:57.631920+0000","last_undegraded":"2024-04-25T11:44:57.631920+0000","last_fullsized":"2024-04-25T11:44:57.631920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"26'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088628+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:44:19.088628+0000","last_peered":"2024-04-25T11:44:19.088628+0000","last_clean":"2024-04-25T11:44:19.088628+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:44:19.088628+0000","last_undegraded":"2024-04-25T11:44:19.088628+0000","last_fullsized":"2024-04-25T11:44:19.088628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.973048+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:44:56.973048+0000","last_peered":"2024-04-25T11:44:56.973048+0000","last_clean":"2024-04-25T11:44:56.973048+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:44:56.973048+0000","last_undegraded":"2024-04-25T11:44:56.973048+0000","last_fullsized":"2024-04-25T11:44:56.973048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'4","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:26.020661+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:44:26.020661+0000","last_peered":"2024-04-25T11:44:26.020661+0000","last_clean":"2024-04-25T11:44:26.020661+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:44:26.020661+0000","last_undegraded":"2024-04-25T11:44:26.020661+0000","last_fullsized":"2024-04-25T11:44:26.020661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.484060+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:44:49.484060+0000","last_peered":"2024-04-25T11:44:49.484060+0000","last_clean":"2024-04-25T11:44:49.484060+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:44:49.484060+0000","last_undegraded":"2024-04-25T11:44:49.484060+0000","last_fullsized":"2024-04-25T11:44:49.484060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'3","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.122357+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:44:19.122357+0000","last_peered":"2024-04-25T11:44:19.122357+0000","last_clean":"2024-04-25T11:44:19.122357+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:44:19.122357+0000","last_undegraded":"2024-04-25T11:44:19.122357+0000","last_fullsized":"2024-04-25T11:44:19.122357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'5","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108672+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:44:19.108672+0000","last_peered":"2024-04-25T11:44:19.108672+0000","last_clean":"2024-04-25T11:44:19.108672+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:44:19.108672+0000","last_undegraded":"2024-04-25T11:44:19.108672+0000","last_fullsized":"2024-04-25T11:44:19.108672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":626661889,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654654,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1030500352,"data_stored":1030500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156529,"ondisk_log_size":156529,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":592,"ondisk_log_size":592,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3395,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3076,"ondisk_log_size":3076,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129712,"kb_used_data":141352,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93242128,"statfs":{"total":96636764160,"available":95479939072,"internally_reserved":0,"allocated":144744448,"data_stored":144477234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1385500,"kb_used_data":202132,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":92986340,"statfs":{"total":96636764160,"available":95218012160,"internally_reserved":0,"allocated":206983168,"data_stored":206731993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575151,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938684,"kb_used_data":162868,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93433156,"statfs":{"total":96636764160,"available":95675551744,"internally_reserved":0,"allocated":166776832,"data_stored":166503551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1153172,"kb_used_data":25740,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93218668,"statfs":{"total":96636764160,"available":95455916032,"internally_reserved":0,"allocated":26357760,"data_stored":26106453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575147,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2136464,"kb_used_data":715976,"kb_used_omap":1,"kb_used_meta":1420478,"kb_avail":92235376,"statfs":{"total":96636764160,"available":94449025024,"internally_reserved":0,"allocated":733159424,"data_stored":732900448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1454569927},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575149,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":853412,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":852414,"kb_avail":93518428,"statfs":{"total":96636764160,"available":95762870272,"internally_reserved":0,"allocated":1011712,"data_stored":748490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872872389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1570232,"kb_used_data":69104,"kb_used_omap":1,"kb_used_meta":1501118,"kb_avail":92801608,"statfs":{"total":96636764160,"available":95028846592,"internally_reserved":0,"allocated":70762496,"data_stored":70506481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1537145286},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575149,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276228,"kb_used_data":225404,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93095612,"statfs":{"total":96636764160,"available":95329906688,"internally_reserved":0,"allocated":230813696,"data_stored":230552518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183062528,"data_stored":183062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61894656,"data_stored":61894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":648699904,"data_stored":648699904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48836608,"data_stored":48836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42930176,"data_stored":42930176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44556288,"data_stored":44556288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:03.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T11:45:03.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:04.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.151+0000 7f0f7564f640 1 Processor -- start 2024-04-25T11:45:04.139 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.151+0000 7f0f7564f640 1 -- start start 2024-04-25T11:45:04.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f7564f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 0x7f0f700ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f7564f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0f700ca010 0x7f0f700ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f7564f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f700cbad0 con 0x7f0f700c8cd0 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f7564f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0f700c8550 con 0x7f0f700ca010 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f7564f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0f700c7c20 con 0x7f0f700ce260 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f67fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 0x7f0f700ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0f700ca010 0x7f0f700ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6effd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0f700c8cd0 0x7f0f7014f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:50374/0 (socket says 172.21.15.136:50374) 2024-04-25T11:45:04.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6effd640 1 -- 172.21.15.136:0/2365645180 learned_addr learned my addr 172.21.15.136:0/2365645180 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:04.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 msgr2=0x7f0f700ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 0x7f0f700ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/2365645180 >> v1:172.21.15.157:6789/0 conn(0x7f0f700c8cd0 legacy=0x7f0f7014f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/2365645180 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f700c7f00 con 0x7f0f700ca010 2024-04-25T11:45:04.142 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.155+0000 7f0f67fff640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 0x7f0f700ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:04.143 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f700ca420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f700c86d0 tx=0x7f0f6801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a51356689cd12a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f6e7fc640 1 -- 172.21.15.136:0/2365645180 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f680b4070 con 0x7f0f700ca010 2024-04-25T11:45:04.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f6e7fc640 1 -- 172.21.15.136:0/2365645180 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f680af600 con 0x7f0f700ca010 2024-04-25T11:45:04.144 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f6e7fc640 1 -- 172.21.15.136:0/2365645180 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f680af9d0 con 0x7f0f700ca010 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 -- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 msgr2=0x7f0f700ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f700ca420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0f700c86d0 tx=0x7f0f6801f130 comp rx=0 tx=0).stop 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 -- 172.21.15.136:0/2365645180 shutdown_connections 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f700ca420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 --2- 172.21.15.136:0/2365645180 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700ce260 0x7f0f700ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 -- 172.21.15.136:0/2365645180 >> 172.21.15.136:0/2365645180 conn(0x7f0f7006f9d0 msgr2=0x7f0f7006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 -- 172.21.15.136:0/2365645180 shutdown_connections 2024-04-25T11:45:04.145 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.159+0000 7f0f7564f640 1 -- 172.21.15.136:0/2365645180 wait complete. 2024-04-25T11:45:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 Processor -- start 2024-04-25T11:45:04.147 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 -- start start 2024-04-25T11:45:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700c8cd0 0x7f0f701f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f7015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0f700cb720 con 0x7f0f700ce260 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0f700ca960 con 0x7f0f700ca010 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f7564f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0f701511a0 con 0x7f0f700c8cd0 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44018/0 (socket says 172.21.15.136:44018) 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/260637590 learned_addr learned my addr 172.21.15.136:0/260637590 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6effd640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700c8cd0 0x7f0f701f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f67fff640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f7015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700c8cd0 msgr2=0x7f0f701f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700c8cd0 0x7f0f701f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 msgr2=0x7f0f7015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f7015a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/260637590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f701f49e0 con 0x7f0f700ce260 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f0f6801cd60 tx=0x7f0f6801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f680b4040 con 0x7f0f700ce260 2024-04-25T11:45:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f680c1070 con 0x7f0f700ce260 2024-04-25T11:45:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.163+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68009450 con 0x7f0f700ce260 2024-04-25T11:45:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7014c6b0 con 0x7f0f700ce260 2024-04-25T11:45:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f701f0b60 con 0x7f0f700ce260 2024-04-25T11:45:04.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7014c400 con 0x7f0f700ce260 2024-04-25T11:45:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0f680ba6b0 con 0x7f0f700ce260 2024-04-25T11:45:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f677fe640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 0x7f0f50071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f0f6812eeb0 con 0x7f0f700ce260 2024-04-25T11:45:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f6effd640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 0x7f0f50071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.167+0000 7f0f6effd640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 0x7f0f50071910 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0f60005890 tx=0x7f0f60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.171+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0f68018c90 con 0x7f0f700ce260 2024-04-25T11:45:04.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.331+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f7006a350 con 0x7f0f5006f400 2024-04-25T11:45:04.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.355+0000 7f0f677fe640 1 -- 172.21.15.136:0/260637590 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334465 (secure 0 0 0) 0x7f0f7006a350 con 0x7f0f5006f400 2024-04-25T11:45:04.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:04.346 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:04.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.359+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 msgr2=0x7f0f50071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.359+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 0x7f0f50071910 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0f60005890 tx=0x7f0f60009040 comp rx=0 tx=0).stop 2024-04-25T11:45:04.347 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.359+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 msgr2=0x7f0f7006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.359+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f0f6801cd60 tx=0x7f0f6801bed0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 shutdown_connections 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f700ce260 0x7f0f7006c840 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f700ca010 0x7f0f7015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f0f5006f400 0x7f0f50071910 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 --2- 172.21.15.136:0/260637590 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f700c8cd0 0x7f0f701f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 >> 172.21.15.136:0/260637590 conn(0x7f0f7006f9d0 msgr2=0x7f0f7014a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 shutdown_connections 2024-04-25T11:45:04.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.363+0000 7f0f7564f640 1 -- 172.21.15.136:0/260637590 wait complete. 2024-04-25T11:45:04.367 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-04-25T11:45:03.415956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642572045,"num_objects":385,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658150,"num_write_kb":15736462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160269,"ondisk_log_size":160269,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":10443404,"kb_used_data":1543564,"kb_used_omap":12,"kb_used_meta":8893619,"kb_avail":744531316,"statfs":{"total":773094113280,"available":762400067584,"internally_reserved":0,"allocated":1580609536,"data_stored":1578527168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9107066418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,13,8,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8339203214,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2050,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003730"},"pg_stats":[{"pgid":"2.1f","version":"40'272","reported_seq":358,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.557577+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:44:53.557577+0000","last_peered":"2024-04-25T11:44:53.557577+0000","last_clean":"2024-04-25T11:44:53.557577+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:44:53.557577+0000","last_undegraded":"2024-04-25T11:44:53.557577+0000","last_fullsized":"2024-04-25T11:44:53.557577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'4","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588204+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:44:28.588204+0000","last_peered":"2024-04-25T11:44:28.588204+0000","last_clean":"2024-04-25T11:44:28.588204+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:44:28.588204+0000","last_undegraded":"2024-04-25T11:44:28.588204+0000","last_fullsized":"2024-04-25T11:44:28.588204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5319","reported_seq":5424,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.505149+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:44:58.505149+0000","last_peered":"2024-04-25T11:44:58.505149+0000","last_clean":"2024-04-25T11:44:58.505149+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:44:58.505149+0000","last_undegraded":"2024-04-25T11:44:58.505149+0000","last_fullsized":"2024-04-25T11:44:58.505149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:22.844221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67156925,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22037,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'9","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:28.588290+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:44:28.588290+0000","last_peered":"2024-04-25T11:44:28.588290+0000","last_clean":"2024-04-25T11:44:28.588290+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:44:28.588290+0000","last_undegraded":"2024-04-25T11:44:28.588290+0000","last_fullsized":"2024-04-25T11:44:28.588290+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:47.051015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.171897+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:44:47.171897+0000","last_peered":"2024-04-25T11:44:47.171897+0000","last_clean":"2024-04-25T11:44:47.171897+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:44:47.171897+0000","last_undegraded":"2024-04-25T11:44:47.171897+0000","last_fullsized":"2024-04-25T11:44:47.171897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4852","reported_seq":4956,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.839347+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:45:00.839347+0000","last_peered":"2024-04-25T11:45:00.839347+0000","last_clean":"2024-04-25T11:45:00.839347+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:00.839347+0000","last_undegraded":"2024-04-25T11:45:00.839347+0000","last_fullsized":"2024-04-25T11:45:00.839347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35243,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20160,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'28","reported_seq":122,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.650684+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:44:42.650684+0000","last_peered":"2024-04-25T11:44:42.650684+0000","last_clean":"2024-04-25T11:44:42.650684+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:42.650684+0000","last_undegraded":"2024-04-25T11:44:42.650684+0000","last_fullsized":"2024-04-25T11:44:42.650684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.299464+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:44:50.299464+0000","last_peered":"2024-04-25T11:44:50.299464+0000","last_clean":"2024-04-25T11:44:50.299464+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:44:50.299464+0000","last_undegraded":"2024-04-25T11:44:50.299464+0000","last_fullsized":"2024-04-25T11:44:50.299464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4262","reported_seq":4357,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.572827+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:45:01.572827+0000","last_peered":"2024-04-25T11:45:01.572827+0000","last_clean":"2024-04-25T11:45:01.572827+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:01.572827+0000","last_undegraded":"2024-04-25T11:45:01.572827+0000","last_fullsized":"2024-04-25T11:45:01.572827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21566,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17219,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108293+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:44:19.108293+0000","last_peered":"2024-04-25T11:44:19.108293+0000","last_clean":"2024-04-25T11:44:19.108293+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:44:19.108293+0000","last_undegraded":"2024-04-25T11:44:19.108293+0000","last_fullsized":"2024-04-25T11:44:19.108293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":62,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.649419+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:45:01.649419+0000","last_peered":"2024-04-25T11:45:01.649419+0000","last_clean":"2024-04-25T11:45:01.649419+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:01.649419+0000","last_undegraded":"2024-04-25T11:45:01.649419+0000","last_fullsized":"2024-04-25T11:45:01.649419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4167","reported_seq":4270,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.384290+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:45:01.384290+0000","last_peered":"2024-04-25T11:45:01.384290+0000","last_clean":"2024-04-25T11:45:01.384290+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:01.384290+0000","last_undegraded":"2024-04-25T11:45:01.384290+0000","last_fullsized":"2024-04-25T11:45:01.384290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16706,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"26'6","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.110341+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:44:19.110341+0000","last_peered":"2024-04-25T11:44:19.110341+0000","last_clean":"2024-04-25T11:44:19.110341+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:44:19.110341+0000","last_undegraded":"2024-04-25T11:44:19.110341+0000","last_fullsized":"2024-04-25T11:44:19.110341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1087","reported_seq":1166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105876+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:44:19.105876+0000","last_peered":"2024-04-25T11:44:19.105876+0000","last_clean":"2024-04-25T11:44:19.105876+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:44:19.105876+0000","last_undegraded":"2024-04-25T11:44:19.105876+0000","last_fullsized":"2024-04-25T11:44:19.105876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5157","reported_seq":5255,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482495+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:45:01.482495+0000","last_peered":"2024-04-25T11:45:01.482495+0000","last_clean":"2024-04-25T11:45:01.482495+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:01.482495+0000","last_undegraded":"2024-04-25T11:45:01.482495+0000","last_fullsized":"2024-04-25T11:45:01.482495+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2946,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21466,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"26'5","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.134609+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:44:19.134609+0000","last_peered":"2024-04-25T11:44:19.134609+0000","last_clean":"2024-04-25T11:44:19.134609+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:44:19.134609+0000","last_undegraded":"2024-04-25T11:44:19.134609+0000","last_fullsized":"2024-04-25T11:44:19.134609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"39'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.124512+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:44:19.124512+0000","last_peered":"2024-04-25T11:44:19.124512+0000","last_clean":"2024-04-25T11:44:19.124512+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:44:19.124512+0000","last_undegraded":"2024-04-25T11:44:19.124512+0000","last_fullsized":"2024-04-25T11:44:19.124512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5165","reported_seq":5262,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.463179+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:45:01.463179+0000","last_peered":"2024-04-25T11:45:01.463179+0000","last_clean":"2024-04-25T11:45:01.463179+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:01.463179+0000","last_undegraded":"2024-04-25T11:45:01.463179+0000","last_fullsized":"2024-04-25T11:45:01.463179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15573,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21560,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.325387+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:44:50.325387+0000","last_peered":"2024-04-25T11:44:50.325387+0000","last_clean":"2024-04-25T11:44:50.325387+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:44:50.325387+0000","last_undegraded":"2024-04-25T11:44:50.325387+0000","last_fullsized":"2024-04-25T11:44:50.325387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111004+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:44:19.111004+0000","last_peered":"2024-04-25T11:44:19.111004+0000","last_clean":"2024-04-25T11:44:19.111004+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:44:19.111004+0000","last_undegraded":"2024-04-25T11:44:19.111004+0000","last_fullsized":"2024-04-25T11:44:19.111004+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6708,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.767277+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:45:01.767277+0000","last_peered":"2024-04-25T11:45:01.767277+0000","last_clean":"2024-04-25T11:45:01.767277+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:01.767277+0000","last_undegraded":"2024-04-25T11:45:01.767277+0000","last_fullsized":"2024-04-25T11:45:01.767277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.688232+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:44:58.688232+0000","last_peered":"2024-04-25T11:44:58.688232+0000","last_clean":"2024-04-25T11:44:58.688232+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:44:58.688232+0000","last_undegraded":"2024-04-25T11:44:58.688232+0000","last_fullsized":"2024-04-25T11:44:58.688232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'6","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105377+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:44:19.105377+0000","last_peered":"2024-04-25T11:44:19.105377+0000","last_clean":"2024-04-25T11:44:19.105377+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:44:19.105377+0000","last_undegraded":"2024-04-25T11:44:19.105377+0000","last_fullsized":"2024-04-25T11:44:19.105377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4791","reported_seq":4887,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.422519+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:45:01.422519+0000","last_peered":"2024-04-25T11:45:01.422519+0000","last_clean":"2024-04-25T11:45:01.422519+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:01.422519+0000","last_undegraded":"2024-04-25T11:45:01.422519+0000","last_fullsized":"2024-04-25T11:45:01.422519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4791,"log_dups_size":0,"ondisk_log_size":4791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8213,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19671,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'17","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.332441+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:44:50.332441+0000","last_peered":"2024-04-25T11:44:50.332441+0000","last_clean":"2024-04-25T11:44:50.332441+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:44:50.332441+0000","last_undegraded":"2024-04-25T11:44:50.332441+0000","last_fullsized":"2024-04-25T11:44:50.332441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'10","reported_seq":97,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111925+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:44:19.111925+0000","last_peered":"2024-04-25T11:44:19.111925+0000","last_clean":"2024-04-25T11:44:19.111925+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:44:19.111925+0000","last_undegraded":"2024-04-25T11:44:19.111925+0000","last_fullsized":"2024-04-25T11:44:19.111925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4481","reported_seq":4582,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.591684+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:45:01.591684+0000","last_peered":"2024-04-25T11:45:01.591684+0000","last_clean":"2024-04-25T11:45:01.591684+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:01.591684+0000","last_undegraded":"2024-04-25T11:45:01.591684+0000","last_fullsized":"2024-04-25T11:45:01.591684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18512,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"26'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.174729+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:44:19.174729+0000","last_peered":"2024-04-25T11:44:19.174729+0000","last_clean":"2024-04-25T11:44:19.174729+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:44:19.174729+0000","last_undegraded":"2024-04-25T11:44:19.174729+0000","last_fullsized":"2024-04-25T11:44:19.174729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'367","reported_seq":447,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.589654+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:44:48.589654+0000","last_peered":"2024-04-25T11:44:48.589654+0000","last_clean":"2024-04-25T11:44:48.589654+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:44:48.589654+0000","last_undegraded":"2024-04-25T11:44:48.589654+0000","last_fullsized":"2024-04-25T11:44:48.589654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4523","reported_seq":4623,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.545359+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:44:58.545359+0000","last_peered":"2024-04-25T11:44:58.545359+0000","last_clean":"2024-04-25T11:44:58.545359+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:44:58.545359+0000","last_undegraded":"2024-04-25T11:44:58.545359+0000","last_fullsized":"2024-04-25T11:44:58.545359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4523,"log_dups_size":0,"ondisk_log_size":4523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:53.486424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58795404,"num_objects":19,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19016,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'7","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.341334+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:44:47.341334+0000","last_peered":"2024-04-25T11:44:47.341334+0000","last_clean":"2024-04-25T11:44:47.341334+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:44:47.341334+0000","last_undegraded":"2024-04-25T11:44:47.341334+0000","last_fullsized":"2024-04-25T11:44:47.341334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'6","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.106657+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:44:19.106657+0000","last_peered":"2024-04-25T11:44:19.106657+0000","last_clean":"2024-04-25T11:44:19.106657+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:19.106657+0000","last_undegraded":"2024-04-25T11:44:19.106657+0000","last_fullsized":"2024-04-25T11:44:19.106657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5066","reported_seq":5166,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.955500+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:44:59.955500+0000","last_peered":"2024-04-25T11:44:59.955500+0000","last_clean":"2024-04-25T11:44:59.955500+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:44:59.955500+0000","last_undegraded":"2024-04-25T11:44:59.955500+0000","last_fullsized":"2024-04-25T11:44:59.955500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16802835,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21087,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'10","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:52.762774+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:52.762774+0000","last_peered":"2024-04-25T11:44:52.762774+0000","last_clean":"2024-04-25T11:44:52.762774+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:44:52.762774+0000","last_undegraded":"2024-04-25T11:44:52.762774+0000","last_fullsized":"2024-04-25T11:44:52.762774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"37'3","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.119332+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:44:19.119332+0000","last_peered":"2024-04-25T11:44:19.119332+0000","last_clean":"2024-04-25T11:44:19.119332+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:44:19.119332+0000","last_undegraded":"2024-04-25T11:44:19.119332+0000","last_fullsized":"2024-04-25T11:44:19.119332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5290","reported_seq":5396,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.948674+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:45:00.948674+0000","last_peered":"2024-04-25T11:45:00.948674+0000","last_clean":"2024-04-25T11:45:00.948674+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:00.948674+0000","last_undegraded":"2024-04-25T11:45:00.948674+0000","last_fullsized":"2024-04-25T11:45:00.948674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20995437,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21937,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.886490+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:44:45.886490+0000","last_peered":"2024-04-25T11:44:45.886490+0000","last_clean":"2024-04-25T11:44:45.886490+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:44:45.886490+0000","last_undegraded":"2024-04-25T11:44:45.886490+0000","last_fullsized":"2024-04-25T11:44:45.886490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'15","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.175082+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:44:24.175082+0000","last_peered":"2024-04-25T11:44:24.175082+0000","last_clean":"2024-04-25T11:44:24.175082+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:44:24.175082+0000","last_undegraded":"2024-04-25T11:44:24.175082+0000","last_fullsized":"2024-04-25T11:44:24.175082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5436","reported_seq":5546,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.076416+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:45:01.076416+0000","last_peered":"2024-04-25T11:45:01.076416+0000","last_clean":"2024-04-25T11:45:01.076416+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:01.076416+0000","last_undegraded":"2024-04-25T11:45:01.076416+0000","last_fullsized":"2024-04-25T11:45:01.076416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4210952,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23006,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"26'9","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.111309+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:44:19.111309+0000","last_peered":"2024-04-25T11:44:19.111309+0000","last_clean":"2024-04-25T11:44:19.111309+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:44:19.111309+0000","last_undegraded":"2024-04-25T11:44:19.111309+0000","last_fullsized":"2024-04-25T11:44:19.111309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'64","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482378+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:45:01.482378+0000","last_peered":"2024-04-25T11:45:01.482378+0000","last_clean":"2024-04-25T11:45:01.482378+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:01.482378+0000","last_undegraded":"2024-04-25T11:45:01.482378+0000","last_fullsized":"2024-04-25T11:45:01.482378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5057","reported_seq":5163,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.917230+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:45:00.917230+0000","last_peered":"2024-04-25T11:45:00.917230+0000","last_clean":"2024-04-25T11:45:00.917230+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:00.917230+0000","last_undegraded":"2024-04-25T11:45:00.917230+0000","last_fullsized":"2024-04-25T11:45:00.917230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4277711,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21666,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'5","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.194821+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:44:47.194821+0000","last_peered":"2024-04-25T11:44:47.194821+0000","last_clean":"2024-04-25T11:44:47.194821+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:44:47.194821+0000","last_undegraded":"2024-04-25T11:44:47.194821+0000","last_fullsized":"2024-04-25T11:44:47.194821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.511395+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:44:46.511395+0000","last_peered":"2024-04-25T11:44:46.511395+0000","last_clean":"2024-04-25T11:44:46.511395+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:44:46.511395+0000","last_undegraded":"2024-04-25T11:44:46.511395+0000","last_fullsized":"2024-04-25T11:44:46.511395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5126","reported_seq":5221,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.068297+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:45:00.068297+0000","last_peered":"2024-04-25T11:45:00.068297+0000","last_clean":"2024-04-25T11:45:00.068297+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:00.068297+0000","last_undegraded":"2024-04-25T11:45:00.068297+0000","last_fullsized":"2024-04-25T11:45:00.068297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16791111,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21514,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"26'3","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.129182+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:44:19.129182+0000","last_peered":"2024-04-25T11:44:19.129182+0000","last_clean":"2024-04-25T11:44:19.129182+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:44:19.129182+0000","last_undegraded":"2024-04-25T11:44:19.129182+0000","last_fullsized":"2024-04-25T11:44:19.129182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'7","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.087665+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:44:19.087665+0000","last_peered":"2024-04-25T11:44:19.087665+0000","last_clean":"2024-04-25T11:44:19.087665+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:44:19.087665+0000","last_undegraded":"2024-04-25T11:44:19.087665+0000","last_fullsized":"2024-04-25T11:44:19.087665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":162,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105537+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:44:19.105537+0000","last_peered":"2024-04-25T11:44:19.105537+0000","last_clean":"2024-04-25T11:44:19.105537+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:44:19.105537+0000","last_undegraded":"2024-04-25T11:44:19.105537+0000","last_fullsized":"2024-04-25T11:44:19.105537+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5031","reported_seq":5126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.959479+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:45:00.959479+0000","last_peered":"2024-04-25T11:45:00.959479+0000","last_clean":"2024-04-25T11:45:00.959479+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:00.959479+0000","last_undegraded":"2024-04-25T11:45:00.959479+0000","last_fullsized":"2024-04-25T11:45:00.959479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4221465,"num_objects":9,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21810,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.613620+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:44:59.613620+0000","last_peered":"2024-04-25T11:44:59.613620+0000","last_clean":"2024-04-25T11:44:59.613620+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:44:59.613620+0000","last_undegraded":"2024-04-25T11:44:59.613620+0000","last_fullsized":"2024-04-25T11:44:59.613620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'16","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.725791+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:45:01.725791+0000","last_peered":"2024-04-25T11:45:01.725791+0000","last_clean":"2024-04-25T11:45:01.725791+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:01.725791+0000","last_undegraded":"2024-04-25T11:45:01.725791+0000","last_fullsized":"2024-04-25T11:45:01.725791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5270","reported_seq":5368,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.055983+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:45:01.055983+0000","last_peered":"2024-04-25T11:45:01.055983+0000","last_clean":"2024-04-25T11:45:01.055983+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:01.055983+0000","last_undegraded":"2024-04-25T11:45:01.055983+0000","last_fullsized":"2024-04-25T11:45:01.055983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54882,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22094,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"26'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088836+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:44:19.088836+0000","last_peered":"2024-04-25T11:44:19.088836+0000","last_clean":"2024-04-25T11:44:19.088836+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:44:19.088836+0000","last_undegraded":"2024-04-25T11:44:19.088836+0000","last_fullsized":"2024-04-25T11:44:19.088836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'13","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006942+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:44:56.006942+0000","last_peered":"2024-04-25T11:44:56.006942+0000","last_clean":"2024-04-25T11:44:56.006942+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:44:56.006942+0000","last_undegraded":"2024-04-25T11:44:56.006942+0000","last_fullsized":"2024-04-25T11:44:56.006942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4937","reported_seq":5037,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.189940+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:45:00.189940+0000","last_peered":"2024-04-25T11:45:00.189940+0000","last_clean":"2024-04-25T11:45:00.189940+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:00.189940+0000","last_undegraded":"2024-04-25T11:45:00.189940+0000","last_fullsized":"2024-04-25T11:45:00.189940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25311976,"num_objects":13,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20945,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.239118+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:44:57.239118+0000","last_peered":"2024-04-25T11:44:57.239118+0000","last_clean":"2024-04-25T11:44:57.239118+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:44:57.239118+0000","last_undegraded":"2024-04-25T11:44:57.239118+0000","last_fullsized":"2024-04-25T11:44:57.239118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'12","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.738638+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:45:01.738638+0000","last_peered":"2024-04-25T11:45:01.738638+0000","last_clean":"2024-04-25T11:45:01.738638+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:01.738638+0000","last_undegraded":"2024-04-25T11:45:01.738638+0000","last_fullsized":"2024-04-25T11:45:01.738638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4817","reported_seq":4920,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.962036+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:45:00.962036+0000","last_peered":"2024-04-25T11:45:00.962036+0000","last_clean":"2024-04-25T11:45:00.962036+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:00.962036+0000","last_undegraded":"2024-04-25T11:45:00.962036+0000","last_fullsized":"2024-04-25T11:45:00.962036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18844,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20361,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:44.444922+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:44:44.444922+0000","last_peered":"2024-04-25T11:44:44.444922+0000","last_clean":"2024-04-25T11:44:44.444922+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:44:44.444922+0000","last_undegraded":"2024-04-25T11:44:44.444922+0000","last_fullsized":"2024-04-25T11:44:44.444922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'5","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.084030+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:44:24.084030+0000","last_peered":"2024-04-25T11:44:24.084030+0000","last_clean":"2024-04-25T11:44:24.084030+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:44:24.084030+0000","last_undegraded":"2024-04-25T11:44:24.084030+0000","last_fullsized":"2024-04-25T11:44:24.084030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4347","reported_seq":4442,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.433320+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:45:01.433320+0000","last_peered":"2024-04-25T11:45:01.433320+0000","last_clean":"2024-04-25T11:45:01.433320+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:01.433320+0000","last_undegraded":"2024-04-25T11:45:01.433320+0000","last_fullsized":"2024-04-25T11:45:01.433320+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2709,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18463,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.271370+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:44:50.271370+0000","last_peered":"2024-04-25T11:44:50.271370+0000","last_clean":"2024-04-25T11:44:50.271370+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:44:50.271370+0000","last_undegraded":"2024-04-25T11:44:50.271370+0000","last_fullsized":"2024-04-25T11:44:50.271370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'31","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732025+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:45:01.732025+0000","last_peered":"2024-04-25T11:45:01.732025+0000","last_clean":"2024-04-25T11:45:01.732025+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:01.732025+0000","last_undegraded":"2024-04-25T11:45:01.732025+0000","last_fullsized":"2024-04-25T11:45:01.732025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5258","reported_seq":5356,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.801479+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:44:59.801479+0000","last_peered":"2024-04-25T11:44:59.801479+0000","last_clean":"2024-04-25T11:44:59.801479+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:44:59.801479+0000","last_undegraded":"2024-04-25T11:44:59.801479+0000","last_fullsized":"2024-04-25T11:44:59.801479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33583936,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21931,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:43.023104+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:44:43.023104+0000","last_peered":"2024-04-25T11:44:43.023104+0000","last_clean":"2024-04-25T11:44:43.023104+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:44:43.023104+0000","last_undegraded":"2024-04-25T11:44:43.023104+0000","last_fullsized":"2024-04-25T11:44:43.023104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'11","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:40.736132+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:44:40.736132+0000","last_peered":"2024-04-25T11:44:40.736132+0000","last_clean":"2024-04-25T11:44:40.736132+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:44:40.736132+0000","last_undegraded":"2024-04-25T11:44:40.736132+0000","last_fullsized":"2024-04-25T11:44:40.736132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5689","reported_seq":5806,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.213416+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:44:58.213416+0000","last_peered":"2024-04-25T11:44:58.213416+0000","last_clean":"2024-04-25T11:44:58.213416+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:44:58.213416+0000","last_undegraded":"2024-04-25T11:44:58.213416+0000","last_fullsized":"2024-04-25T11:44:58.213416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:21.933718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88343888,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23822,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"26'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105615+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:44:19.105615+0000","last_peered":"2024-04-25T11:44:19.105615+0000","last_clean":"2024-04-25T11:44:19.105615+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:44:19.105615+0000","last_undegraded":"2024-04-25T11:44:19.105615+0000","last_fullsized":"2024-04-25T11:44:19.105615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"39'9","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088317+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:44:19.088317+0000","last_peered":"2024-04-25T11:44:19.088317+0000","last_clean":"2024-04-25T11:44:19.088317+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:44:19.088317+0000","last_undegraded":"2024-04-25T11:44:19.088317+0000","last_fullsized":"2024-04-25T11:44:19.088317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4245","reported_seq":4353,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.531793+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:44:58.531793+0000","last_peered":"2024-04-25T11:44:58.531793+0000","last_clean":"2024-04-25T11:44:58.531793+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:44:58.531793+0000","last_undegraded":"2024-04-25T11:44:58.531793+0000","last_fullsized":"2024-04-25T11:44:58.531793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58753411,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17543,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.027133+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:44:55.027133+0000","last_peered":"2024-04-25T11:44:55.027133+0000","last_clean":"2024-04-25T11:44:55.027133+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:44:55.027133+0000","last_undegraded":"2024-04-25T11:44:55.027133+0000","last_fullsized":"2024-04-25T11:44:55.027133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'10","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.512801+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:44:46.512801+0000","last_peered":"2024-04-25T11:44:46.512801+0000","last_clean":"2024-04-25T11:44:46.512801+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:44:46.512801+0000","last_undegraded":"2024-04-25T11:44:46.512801+0000","last_fullsized":"2024-04-25T11:44:46.512801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4278","reported_seq":4372,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.989857+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:45:00.989857+0000","last_peered":"2024-04-25T11:45:00.989857+0000","last_clean":"2024-04-25T11:45:00.989857+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:00.989857+0000","last_undegraded":"2024-04-25T11:45:00.989857+0000","last_fullsized":"2024-04-25T11:45:00.989857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17948,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.022826+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:44:45.022826+0000","last_peered":"2024-04-25T11:44:45.022826+0000","last_clean":"2024-04-25T11:44:45.022826+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:44:45.022826+0000","last_undegraded":"2024-04-25T11:44:45.022826+0000","last_fullsized":"2024-04-25T11:44:45.022826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'348","reported_seq":441,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.649851+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:44:42.649851+0000","last_peered":"2024-04-25T11:44:42.649851+0000","last_clean":"2024-04-25T11:44:42.649851+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:44:42.649851+0000","last_undegraded":"2024-04-25T11:44:42.649851+0000","last_fullsized":"2024-04-25T11:44:42.649851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4214","reported_seq":4312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:00.997646+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:45:00.997646+0000","last_peered":"2024-04-25T11:45:00.997646+0000","last_clean":"2024-04-25T11:45:00.997646+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:00.997646+0000","last_undegraded":"2024-04-25T11:45:00.997646+0000","last_fullsized":"2024-04-25T11:45:00.997646+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240135,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17728,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'8","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.051954+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:44:58.051954+0000","last_peered":"2024-04-25T11:44:58.051954+0000","last_clean":"2024-04-25T11:44:58.051954+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:44:58.051954+0000","last_undegraded":"2024-04-25T11:44:58.051954+0000","last_fullsized":"2024-04-25T11:44:58.051954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'28","reported_seq":103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006617+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:44:56.006617+0000","last_peered":"2024-04-25T11:44:56.006617+0000","last_clean":"2024-04-25T11:44:56.006617+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:44:56.006617+0000","last_undegraded":"2024-04-25T11:44:56.006617+0000","last_fullsized":"2024-04-25T11:44:56.006617+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":31,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4469","reported_seq":4558,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.583673+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:45:01.583673+0000","last_peered":"2024-04-25T11:45:01.583673+0000","last_clean":"2024-04-25T11:45:01.583673+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:01.583673+0000","last_undegraded":"2024-04-25T11:45:01.583673+0000","last_fullsized":"2024-04-25T11:45:01.583673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410839,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18507,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.636324+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:44:50.636324+0000","last_peered":"2024-04-25T11:44:50.636324+0000","last_clean":"2024-04-25T11:44:50.636324+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:44:50.636324+0000","last_undegraded":"2024-04-25T11:44:50.636324+0000","last_fullsized":"2024-04-25T11:44:50.636324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.089262+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:44:19.089262+0000","last_peered":"2024-04-25T11:44:19.089262+0000","last_clean":"2024-04-25T11:44:19.089262+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:44:19.089262+0000","last_undegraded":"2024-04-25T11:44:19.089262+0000","last_fullsized":"2024-04-25T11:44:19.089262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4316","reported_seq":4419,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.854217+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:44:58.854217+0000","last_peered":"2024-04-25T11:44:58.854217+0000","last_clean":"2024-04-25T11:44:58.854217+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:44:58.854217+0000","last_undegraded":"2024-04-25T11:44:58.854217+0000","last_fullsized":"2024-04-25T11:44:58.854217+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:11:35.278403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75500283,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17930,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.055161+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:44:58.055161+0000","last_peered":"2024-04-25T11:44:58.055161+0000","last_clean":"2024-04-25T11:44:58.055161+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:44:58.055161+0000","last_undegraded":"2024-04-25T11:44:58.055161+0000","last_fullsized":"2024-04-25T11:44:58.055161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'14","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.755327+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:44:56.755327+0000","last_peered":"2024-04-25T11:44:56.755327+0000","last_clean":"2024-04-25T11:44:56.755327+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:44:56.755327+0000","last_undegraded":"2024-04-25T11:44:56.755327+0000","last_fullsized":"2024-04-25T11:44:56.755327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3766","reported_seq":3868,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.789112+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:44:58.789112+0000","last_peered":"2024-04-25T11:44:58.789112+0000","last_clean":"2024-04-25T11:44:58.789112+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:44:58.789112+0000","last_undegraded":"2024-04-25T11:44:58.789112+0000","last_fullsized":"2024-04-25T11:44:58.789112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75515216,"num_objects":23,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15950,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5407","reported_seq":5520,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.592390+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:44:59.592390+0000","last_peered":"2024-04-25T11:44:59.592390+0000","last_clean":"2024-04-25T11:44:59.592390+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:44:59.592390+0000","last_undegraded":"2024-04-25T11:44:59.592390+0000","last_fullsized":"2024-04-25T11:44:59.592390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16805846,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22729,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'4","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.042946+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:44:46.042946+0000","last_peered":"2024-04-25T11:44:46.042946+0000","last_clean":"2024-04-25T11:44:46.042946+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:44:46.042946+0000","last_undegraded":"2024-04-25T11:44:46.042946+0000","last_fullsized":"2024-04-25T11:44:46.042946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'12","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.021180+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:44:31.021180+0000","last_peered":"2024-04-25T11:44:31.021180+0000","last_clean":"2024-04-25T11:44:31.021180+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:44:31.021180+0000","last_undegraded":"2024-04-25T11:44:31.021180+0000","last_fullsized":"2024-04-25T11:44:31.021180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5672","reported_seq":5787,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.033114+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:45:01.033114+0000","last_peered":"2024-04-25T11:45:01.033114+0000","last_clean":"2024-04-25T11:45:01.033114+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:01.033114+0000","last_undegraded":"2024-04-25T11:45:01.033114+0000","last_fullsized":"2024-04-25T11:45:01.033114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8447109,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24242,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:33.648968+0000","last_change":"2024-04-25T11:20:20.037148+0000","last_active":"2024-04-25T11:44:33.648968+0000","last_peered":"2024-04-25T11:44:33.648968+0000","last_clean":"2024-04-25T11:44:33.648968+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:44:33.648968+0000","last_undegraded":"2024-04-25T11:44:33.648968+0000","last_fullsized":"2024-04-25T11:44:33.648968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.507269+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:44:46.507269+0000","last_peered":"2024-04-25T11:44:46.507269+0000","last_clean":"2024-04-25T11:44:46.507269+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:44:46.507269+0000","last_undegraded":"2024-04-25T11:44:46.507269+0000","last_fullsized":"2024-04-25T11:44:46.507269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"38'11","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.510877+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:44:46.510877+0000","last_peered":"2024-04-25T11:44:46.510877+0000","last_clean":"2024-04-25T11:44:46.510877+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:44:46.510877+0000","last_undegraded":"2024-04-25T11:44:46.510877+0000","last_fullsized":"2024-04-25T11:44:46.510877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.709218+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:44:59.709218+0000","last_peered":"2024-04-25T11:44:59.709218+0000","last_clean":"2024-04-25T11:44:59.709218+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:44:59.709218+0000","last_undegraded":"2024-04-25T11:44:59.709218+0000","last_fullsized":"2024-04-25T11:44:59.709218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.368416+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:45:01.368416+0000","last_peered":"2024-04-25T11:45:01.368416+0000","last_clean":"2024-04-25T11:45:01.368416+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:01.368416+0000","last_undegraded":"2024-04-25T11:45:01.368416+0000","last_fullsized":"2024-04-25T11:45:01.368416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'636","reported_seq":710,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:46.539919+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:44:46.539919+0000","last_peered":"2024-04-25T11:44:46.539919+0000","last_clean":"2024-04-25T11:44:46.539919+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:44:46.539919+0000","last_undegraded":"2024-04-25T11:44:46.539919+0000","last_fullsized":"2024-04-25T11:44:46.539919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'10","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.315631+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:44:50.315631+0000","last_peered":"2024-04-25T11:44:50.315631+0000","last_clean":"2024-04-25T11:44:50.315631+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:44:50.315631+0000","last_undegraded":"2024-04-25T11:44:50.315631+0000","last_fullsized":"2024-04-25T11:44:50.315631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5248","reported_seq":5351,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.821173+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:44:58.821173+0000","last_peered":"2024-04-25T11:44:58.821173+0000","last_clean":"2024-04-25T11:44:58.821173+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:44:58.821173+0000","last_undegraded":"2024-04-25T11:44:58.821173+0000","last_fullsized":"2024-04-25T11:44:58.821173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5248,"log_dups_size":0,"ondisk_log_size":5248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:25.633120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46139613,"num_objects":15,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22077,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.073823+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:44:53.073823+0000","last_peered":"2024-04-25T11:44:53.073823+0000","last_clean":"2024-04-25T11:44:53.073823+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:44:53.073823+0000","last_undegraded":"2024-04-25T11:44:53.073823+0000","last_fullsized":"2024-04-25T11:44:53.073823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.606903+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:45:01.606903+0000","last_peered":"2024-04-25T11:45:01.606903+0000","last_clean":"2024-04-25T11:45:01.606903+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:01.606903+0000","last_undegraded":"2024-04-25T11:45:01.606903+0000","last_fullsized":"2024-04-25T11:45:01.606903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'14","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.513817+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:44:54.513817+0000","last_peered":"2024-04-25T11:44:54.513817+0000","last_clean":"2024-04-25T11:44:54.513817+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:44:54.513817+0000","last_undegraded":"2024-04-25T11:44:54.513817+0000","last_fullsized":"2024-04-25T11:44:54.513817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'7","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.120881+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:44:19.120881+0000","last_peered":"2024-04-25T11:44:19.120881+0000","last_clean":"2024-04-25T11:44:19.120881+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:44:19.120881+0000","last_undegraded":"2024-04-25T11:44:19.120881+0000","last_fullsized":"2024-04-25T11:44:19.120881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.338193+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:44:50.338193+0000","last_peered":"2024-04-25T11:44:50.338193+0000","last_clean":"2024-04-25T11:44:50.338193+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:44:50.338193+0000","last_undegraded":"2024-04-25T11:44:50.338193+0000","last_fullsized":"2024-04-25T11:44:50.338193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.012448+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:44:56.012448+0000","last_peered":"2024-04-25T11:44:56.012448+0000","last_clean":"2024-04-25T11:44:56.012448+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:56.012448+0000","last_undegraded":"2024-04-25T11:44:56.012448+0000","last_fullsized":"2024-04-25T11:44:56.012448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.845960+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:44:58.845960+0000","last_peered":"2024-04-25T11:44:58.845960+0000","last_clean":"2024-04-25T11:44:58.845960+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:44:58.845960+0000","last_undegraded":"2024-04-25T11:44:58.845960+0000","last_fullsized":"2024-04-25T11:44:58.845960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.266380+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:44:50.266380+0000","last_peered":"2024-04-25T11:44:50.266380+0000","last_clean":"2024-04-25T11:44:50.266380+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:44:50.266380+0000","last_undegraded":"2024-04-25T11:44:50.266380+0000","last_fullsized":"2024-04-25T11:44:50.266380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.291749+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:44:50.291749+0000","last_peered":"2024-04-25T11:44:50.291749+0000","last_clean":"2024-04-25T11:44:50.291749+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:44:50.291749+0000","last_undegraded":"2024-04-25T11:44:50.291749+0000","last_fullsized":"2024-04-25T11:44:50.291749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'12","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.021270+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:44:47.021270+0000","last_peered":"2024-04-25T11:44:47.021270+0000","last_clean":"2024-04-25T11:44:47.021270+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:44:47.021270+0000","last_undegraded":"2024-04-25T11:44:47.021270+0000","last_fullsized":"2024-04-25T11:44:47.021270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:48.361691+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:44:48.361691+0000","last_peered":"2024-04-25T11:44:48.361691+0000","last_clean":"2024-04-25T11:44:48.361691+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:44:48.361691+0000","last_undegraded":"2024-04-25T11:44:48.361691+0000","last_fullsized":"2024-04-25T11:44:48.361691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.848357+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:44:54.848357+0000","last_peered":"2024-04-25T11:44:54.848357+0000","last_clean":"2024-04-25T11:44:54.848357+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:44:54.848357+0000","last_undegraded":"2024-04-25T11:44:54.848357+0000","last_fullsized":"2024-04-25T11:44:54.848357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.113943+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:44:19.113943+0000","last_peered":"2024-04-25T11:44:19.113943+0000","last_clean":"2024-04-25T11:44:19.113943+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:44:19.113943+0000","last_undegraded":"2024-04-25T11:44:19.113943+0000","last_fullsized":"2024-04-25T11:44:19.113943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'12","reported_seq":60,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:51.941786+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:44:51.941786+0000","last_peered":"2024-04-25T11:44:51.941786+0000","last_clean":"2024-04-25T11:44:51.941786+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:44:51.941786+0000","last_undegraded":"2024-04-25T11:44:51.941786+0000","last_fullsized":"2024-04-25T11:44:51.941786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732164+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:45:01.732164+0000","last_peered":"2024-04-25T11:45:01.732164+0000","last_clean":"2024-04-25T11:45:01.732164+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:01.732164+0000","last_undegraded":"2024-04-25T11:45:01.732164+0000","last_fullsized":"2024-04-25T11:45:01.732164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.825982+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:44:56.825982+0000","last_peered":"2024-04-25T11:44:56.825982+0000","last_clean":"2024-04-25T11:44:56.825982+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:44:56.825982+0000","last_undegraded":"2024-04-25T11:44:56.825982+0000","last_fullsized":"2024-04-25T11:44:56.825982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:55.844083+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:44:55.844083+0000","last_peered":"2024-04-25T11:44:55.844083+0000","last_clean":"2024-04-25T11:44:55.844083+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:44:55.844083+0000","last_undegraded":"2024-04-25T11:44:55.844083+0000","last_fullsized":"2024-04-25T11:44:55.844083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.240393+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:44:49.240393+0000","last_peered":"2024-04-25T11:44:49.240393+0000","last_clean":"2024-04-25T11:44:49.240393+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:44:49.240393+0000","last_undegraded":"2024-04-25T11:44:49.240393+0000","last_fullsized":"2024-04-25T11:44:49.240393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.999436+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:44:56.999436+0000","last_peered":"2024-04-25T11:44:56.999436+0000","last_clean":"2024-04-25T11:44:56.999436+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:44:56.999436+0000","last_undegraded":"2024-04-25T11:44:56.999436+0000","last_fullsized":"2024-04-25T11:44:56.999436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'10","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.496713+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:44:50.496713+0000","last_peered":"2024-04-25T11:44:50.496713+0000","last_clean":"2024-04-25T11:44:50.496713+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:44:50.496713+0000","last_undegraded":"2024-04-25T11:44:50.496713+0000","last_fullsized":"2024-04-25T11:44:50.496713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'9","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:47.323680+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:44:47.323680+0000","last_peered":"2024-04-25T11:44:47.323680+0000","last_clean":"2024-04-25T11:44:47.323680+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:44:47.323680+0000","last_undegraded":"2024-04-25T11:44:47.323680+0000","last_fullsized":"2024-04-25T11:44:47.323680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137154+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:44:31.137154+0000","last_peered":"2024-04-25T11:44:31.137154+0000","last_clean":"2024-04-25T11:44:31.137154+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:44:31.137154+0000","last_undegraded":"2024-04-25T11:44:31.137154+0000","last_fullsized":"2024-04-25T11:44:31.137154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'10","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137174+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:44:31.137174+0000","last_peered":"2024-04-25T11:44:31.137174+0000","last_clean":"2024-04-25T11:44:31.137174+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:44:31.137174+0000","last_undegraded":"2024-04-25T11:44:31.137174+0000","last_fullsized":"2024-04-25T11:44:31.137174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'14","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.177913+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:44:56.177913+0000","last_peered":"2024-04-25T11:44:56.177913+0000","last_clean":"2024-04-25T11:44:56.177913+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:44:56.177913+0000","last_undegraded":"2024-04-25T11:44:56.177913+0000","last_fullsized":"2024-04-25T11:44:56.177913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'5","reported_seq":65,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.663176+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:44:53.663176+0000","last_peered":"2024-04-25T11:44:53.663176+0000","last_clean":"2024-04-25T11:44:53.663176+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:44:53.663176+0000","last_undegraded":"2024-04-25T11:44:53.663176+0000","last_fullsized":"2024-04-25T11:44:53.663176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'6","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.631920+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:44:57.631920+0000","last_peered":"2024-04-25T11:44:57.631920+0000","last_clean":"2024-04-25T11:44:57.631920+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:44:57.631920+0000","last_undegraded":"2024-04-25T11:44:57.631920+0000","last_fullsized":"2024-04-25T11:44:57.631920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"26'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088628+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:44:19.088628+0000","last_peered":"2024-04-25T11:44:19.088628+0000","last_clean":"2024-04-25T11:44:19.088628+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:44:19.088628+0000","last_undegraded":"2024-04-25T11:44:19.088628+0000","last_fullsized":"2024-04-25T11:44:19.088628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'7","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.973048+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:44:56.973048+0000","last_peered":"2024-04-25T11:44:56.973048+0000","last_clean":"2024-04-25T11:44:56.973048+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:44:56.973048+0000","last_undegraded":"2024-04-25T11:44:56.973048+0000","last_fullsized":"2024-04-25T11:44:56.973048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'4","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:26.020661+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:44:26.020661+0000","last_peered":"2024-04-25T11:44:26.020661+0000","last_clean":"2024-04-25T11:44:26.020661+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:44:26.020661+0000","last_undegraded":"2024-04-25T11:44:26.020661+0000","last_fullsized":"2024-04-25T11:44:26.020661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:49.484060+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:44:49.484060+0000","last_peered":"2024-04-25T11:44:49.484060+0000","last_clean":"2024-04-25T11:44:49.484060+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:44:49.484060+0000","last_undegraded":"2024-04-25T11:44:49.484060+0000","last_fullsized":"2024-04-25T11:44:49.484060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'3","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.122357+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:44:19.122357+0000","last_peered":"2024-04-25T11:44:19.122357+0000","last_clean":"2024-04-25T11:44:19.122357+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:44:19.122357+0000","last_undegraded":"2024-04-25T11:44:19.122357+0000","last_fullsized":"2024-04-25T11:44:19.122357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'5","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108672+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:44:19.108672+0000","last_peered":"2024-04-25T11:44:19.108672+0000","last_clean":"2024-04-25T11:44:19.108672+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:44:19.108672+0000","last_undegraded":"2024-04-25T11:44:19.108672+0000","last_fullsized":"2024-04-25T11:44:19.108672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":626661889,"num_objects":281,"num_object_clones":0,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654654,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1030500352,"data_stored":1030500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156529,"ondisk_log_size":156529,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":592,"ondisk_log_size":592,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3395,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3076,"ondisk_log_size":3076,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129712,"kb_used_data":141352,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93242128,"statfs":{"total":96636764160,"available":95479939072,"internally_reserved":0,"allocated":144744448,"data_stored":144477234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1385500,"kb_used_data":202132,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":92986340,"statfs":{"total":96636764160,"available":95218012160,"internally_reserved":0,"allocated":206983168,"data_stored":206731993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575151,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":938684,"kb_used_data":162868,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93433156,"statfs":{"total":96636764160,"available":95675551744,"internally_reserved":0,"allocated":166776832,"data_stored":166503551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1153172,"kb_used_data":25740,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93218668,"statfs":{"total":96636764160,"available":95455916032,"internally_reserved":0,"allocated":26357760,"data_stored":26106453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575147,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2136464,"kb_used_data":715976,"kb_used_omap":1,"kb_used_meta":1420478,"kb_avail":92235376,"statfs":{"total":96636764160,"available":94449025024,"internally_reserved":0,"allocated":733159424,"data_stored":732900448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1454569927},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575149,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":853412,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":852414,"kb_avail":93518428,"statfs":{"total":96636764160,"available":95762870272,"internally_reserved":0,"allocated":1011712,"data_stored":748490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872872389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1570232,"kb_used_data":69104,"kb_used_omap":1,"kb_used_meta":1501118,"kb_avail":92801608,"statfs":{"total":96636764160,"available":95028846592,"internally_reserved":0,"allocated":70762496,"data_stored":70506481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1537145286},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575149,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1276228,"kb_used_data":225404,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93095612,"statfs":{"total":96636764160,"available":95329906688,"internally_reserved":0,"allocated":230813696,"data_stored":230552518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183062528,"data_stored":183062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61894656,"data_stored":61894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":648699904,"data_stored":648699904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48836608,"data_stored":48836608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42930176,"data_stored":42930176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44556288,"data_stored":44556288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:04.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:45:04.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.563+0000 7faa0f68c640 1 Processor -- start 2024-04-25T11:45:04.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.563+0000 7faa0f68c640 1 -- start start 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0f68c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faa080cf490 0x7faa080cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0f68c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0815c410 con 0x7faa0806e470 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0f68c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa080cfde0 con 0x7faa08156750 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0f68c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faa0815c6b0 con 0x7faa080cf490 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0dc29640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faa0806e470 0x7faa0806e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:55554/0 (socket says 172.21.15.136:55554) 2024-04-25T11:45:04.552 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0dc29640 1 -- 172.21.15.136:0/1582715443 learned_addr learned my addr 172.21.15.136:0/1582715443 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:04.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401160940 0 0) 0x7faa0815c410 con 0x7faa0806e470 2024-04-25T11:45:04.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9f4003530 con 0x7faa0806e470 2024-04-25T11:45:04.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 --2- 172.21.15.136:0/1582715443 >> v2:172.21.15.157:3300/0 conn(0x7faa080cf490 0x7faa080cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 425784516 0 0) 0x7fa9f4003530 con 0x7faa0806e470 2024-04-25T11:45:04.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 >> v2:172.21.15.157:3300/0 conn(0x7faa080cf490 msgr2=0x7faa080cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 --2- 172.21.15.136:0/1582715443 >> v2:172.21.15.157:3300/0 conn(0x7faa080cf490 0x7faa080cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 >> v1:172.21.15.157:6789/0 conn(0x7faa08156750 legacy=0x7faa080ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0815c990 con 0x7faa0806e470 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7faa00001a00 con 0x7faa0806e470 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 >> v1:172.21.15.136:6789/0 conn(0x7faa0806e470 legacy=0x7faa0806e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9f40039c0 0x7fa9f4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7fa9f4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9f40051d0 0x7fa9f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faa0815c6b0 con 0x7fa9f4004810 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f400c170 con 0x7fa9f40051d0 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f400c370 con 0x7fa9f40039c0 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7fa9f4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0dc29640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9f40051d0 0x7fa9f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0d428640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9f40039c0 0x7fa9f4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 -- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9f40039c0 msgr2=0x7fa9f4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9f40039c0 0x7fa9f4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 -- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9f40051d0 msgr2=0x7fa9f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9f40051d0 0x7fa9f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0cc27640 1 -- 172.21.15.136:0/1582715443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f4003530 con 0x7fa9f4004810 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0d428640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa9f40039c0 0x7fa9f4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0dc29640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa9f40051d0 0x7fa9f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.567+0000 7faa0dc29640 1 -- 172.21.15.136:0/1582715443 reap_dead start 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0cc27640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7fa9f4004c20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ab0 tx=0x7faa0401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155c0982c579206e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040b2070 con 0x7fa9f4004810 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa040a5640 con 0x7fa9f4004810 2024-04-25T11:45:04.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7fa9fffff640 1 -- 172.21.15.136:0/1582715443 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040096b0 con 0x7fa9f4004810 2024-04-25T11:45:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 msgr2=0x7fa9f4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7fa9f4004c20 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7faa04009ab0 tx=0x7faa0401ca50 comp rx=0 tx=0).stop 2024-04-25T11:45:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- 172.21.15.136:0/1582715443 shutdown_connections 2024-04-25T11:45:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 --2- 172.21.15.136:0/1582715443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7fa9f4004c20 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- 172.21.15.136:0/1582715443 >> 172.21.15.136:0/1582715443 conn(0x7faa08070f70 msgr2=0x7faa08068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- 172.21.15.136:0/1582715443 shutdown_connections 2024-04-25T11:45:04.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- 172.21.15.136:0/1582715443 wait complete. 2024-04-25T11:45:04.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 Processor -- start 2024-04-25T11:45:04.559 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.571+0000 7faa0f68c640 1 -- start start 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa080b3ea0 0x7faa0815af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 0x7faa08151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faa0806d860 con 0x7fa9f4004810 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faa0815cf50 con 0x7faa080b3ea0 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faa0806d9e0 con 0x7faa0815b4b0 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44044/0 (socket says 172.21.15.136:44044) 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 -- 172.21.15.136:0/3640020279 learned_addr learned my addr 172.21.15.136:0/3640020279 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0cc27640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa080b3ea0 0x7faa0815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0dc29640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 0x7faa08151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 -- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 msgr2=0x7faa08151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 0x7faa08151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 -- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa080b3ea0 msgr2=0x7faa0815af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa080b3ea0 0x7faa0815af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 -- 172.21.15.136:0/3640020279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa080af800 con 0x7fa9f4004810 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0dc29640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 0x7faa08151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0d428640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800d770 tx=0x7fa9f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8016070 con 0x7fa9f4004810 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f800b8b0 con 0x7fa9f4004810 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa081f3030 con 0x7fa9f4004810 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f800bc80 con 0x7fa9f4004810 2024-04-25T11:45:04.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa081f2d50 con 0x7fa9f4004810 2024-04-25T11:45:04.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.575+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa080af9d0 con 0x7fa9f4004810 2024-04-25T11:45:04.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.579+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa9f8019070 con 0x7fa9f4004810 2024-04-25T11:45:04.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.579+0000 7fa9fdffb640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 0x7fa9ec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:04.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.579+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fa9f808feb0 con 0x7fa9f4004810 2024-04-25T11:45:04.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.579+0000 7faa0cc27640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 0x7fa9ec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:04.564 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.579+0000 7faa0cc27640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 0x7fa9ec071940 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faa040af000 tx=0x7faa04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:04.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.583+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa9f8020020 con 0x7fa9f4004810 2024-04-25T11:45:04.732 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.747+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa0806a350 con 0x7fa9f4004810 2024-04-25T11:45:04.733 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.747+0000 7fa9fdffb640 1 -- 172.21.15.136:0/3640020279 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+20778 (secure 0 0 0) 0x7fa9f800be20 con 0x7fa9f4004810 2024-04-25T11:45:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:04.733 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:44:19.077280+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6837/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6836/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6834/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6838/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6835/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6839/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6833/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.136:6836/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.136:6832/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6834/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6832/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.157:6839/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6834/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.136:6836/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.136:6837/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.157:6838/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6837/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.157:6835/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6835/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6833/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6837/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6835/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.157:6834/694883637":"2024-04-26T11:30:48.887575+0000","172.21.15.157:6836/694883637":"2024-04-26T11:30:48.887575+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:45:04.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 msgr2=0x7fa9ec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.735 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 0x7fa9ec071940 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faa040af000 tx=0x7faa04008040 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 msgr2=0x7faa080b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800d770 tx=0x7fa9f800dc30 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 shutdown_connections 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fa9ec06f430 0x7fa9ec071940 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa0815b4b0 0x7faa08151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa080b3ea0 0x7faa0815af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 --2- 172.21.15.136:0/3640020279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9f4004810 0x7faa080b4d80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 >> 172.21.15.136:0/3640020279 conn(0x7faa08070f70 msgr2=0x7faa080cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 shutdown_connections 2024-04-25T11:45:04.736 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:04.751+0000 7faa0f68c640 1 -- 172.21.15.136:0/3640020279 wait complete. 2024-04-25T11:45:05.755 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T11:45:05.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:05.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.915+0000 7fd2505e1640 1 Processor -- start 2024-04-25T11:45:05.902 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.915+0000 7fd2505e1640 1 -- start start 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480c8f50 0x7fd248152e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd248158d20 0x7fd248159130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2480c5040 0x7fd2480c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fd2480c7d10 con 0x7fd2480c5040 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd2480ca1f0 con 0x7fd248158d20 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd2505e1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2480c73e0 con 0x7fd2480c8f50 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2480c5040 0x7fd2480c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2480c5040 0x7fd2480c5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44590/0 (socket says 172.21.15.136:44590) 2024-04-25T11:45:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 -- 172.21.15.136:0/3599168837 learned_addr learned my addr 172.21.15.136:0/3599168837 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24e37d640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480c8f50 0x7fd248152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24db7c640 1 --2- 172.21.15.136:0/3599168837 >> v2:172.21.15.157:3300/0 conn(0x7fd248158d20 0x7fd248159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 -- 172.21.15.136:0/3599168837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480c8f50 msgr2=0x7fd248152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480c8f50 0x7fd248152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 -- 172.21.15.136:0/3599168837 >> v2:172.21.15.157:3300/0 conn(0x7fd248158d20 msgr2=0x7fd248159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/3599168837 >> v2:172.21.15.157:3300/0 conn(0x7fd248158d20 0x7fd248159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.906 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 -- 172.21.15.136:0/3599168837 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2480c76c0 con 0x7fd2480c5040 2024-04-25T11:45:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd2480c5450 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd2480cebb0 tx=0x7fd23801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f49841749d8e919 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:05.907 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24d37b640 1 -- 172.21.15.136:0/3599168837 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2380b4070 con 0x7fd2480c5040 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24d37b640 1 -- 172.21.15.136:0/3599168837 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2380af400 con 0x7fd2480c5040 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.919+0000 7fd24d37b640 1 -- 172.21.15.136:0/3599168837 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2380af7d0 con 0x7fd2480c5040 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- 172.21.15.136:0/3599168837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 msgr2=0x7fd2480c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd2480c5450 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd2480cebb0 tx=0x7fd23801f1e0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- 172.21.15.136:0/3599168837 shutdown_connections 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd2480c5450 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- 172.21.15.136:0/3599168837 >> v2:172.21.15.157:3300/0 conn(0x7fd248158d20 0x7fd248159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- 172.21.15.136:0/3599168837 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480c8f50 0x7fd248152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- 172.21.15.136:0/3599168837 >> 172.21.15.136:0/3599168837 conn(0x7fd2480cdb40 msgr2=0x7fd2480cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- 172.21.15.136:0/3599168837 shutdown_connections 2024-04-25T11:45:05.908 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- 172.21.15.136:0/3599168837 wait complete. 2024-04-25T11:45:05.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 Processor -- start 2024-04-25T11:45:05.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- start start 2024-04-25T11:45:05.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd24814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.911 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480c8f50 0x7fd248147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248158d20 0x7fd248148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd248153370 con 0x7fd2480c5040 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2480c9ed0 con 0x7fd2480c8f50 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd2505e1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd248159670 con 0x7fd248158d20 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd24e37d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd24814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd24e37d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd24814eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44598/0 (socket says 172.21.15.136:44598) 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.923+0000 7fd24e37d640 1 -- 172.21.15.136:0/439154788 learned_addr learned my addr 172.21.15.136:0/439154788 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248158d20 0x7fd248148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24db7c640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480c8f50 0x7fd248147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.912 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 -- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248158d20 msgr2=0x7fd248148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248158d20 0x7fd248148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 -- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480c8f50 msgr2=0x7fd248147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480c8f50 0x7fd248147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24814f060 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24e37d640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd24814eb20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd24000d5f0 tx=0x7fd24000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2400125f0 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd240019070 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481580f0 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2400129b0 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2481586b0 con 0x7fd2480c5040 2024-04-25T11:45:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd24806a3c0 con 0x7fd2480c5040 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd240012b50 con 0x7fd2480c5040 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd23406f400 0x7fd234071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fd240090a60 con 0x7fd2480c5040 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fd234079800 0x7fd23407bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd23407c3c0 con 0x7fd234079800 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd240090ef0 con 0x7fd2480c5040 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24db7c640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd23406f400 0x7fd234071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fd234079800 0x7fd23407bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.927+0000 7fd24eb7e640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fd234079800 0x7fd23407bcf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.931+0000 7fd24db7c640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd23406f400 0x7fd234071910 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd2481556d0 tx=0x7fd23c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.931+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== osd.0 v2:172.21.15.136:6800/3420887400 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd23407c3c0 con 0x7fd234079800 2024-04-25T11:45:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 --> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd248155700 con 0x7fd234079800 2024-04-25T11:45:05.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd246ffd640 1 -- 172.21.15.136:0/439154788 <== osd.0 v2:172.21.15.136:6800/3420887400 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd248155700 con 0x7fd234079800 2024-04-25T11:45:05.928 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:05.928 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:05.928 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fd234079800 msgr2=0x7fd23407bcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:6800/3420887400,v1:172.21.15.136:6801/3420887400] conn(0x7fd234079800 0x7fd23407bcf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd23406f400 msgr2=0x7fd234071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd23406f400 0x7fd234071910 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd2481556d0 tx=0x7fd23c008040 comp rx=0 tx=0).stop 2024-04-25T11:45:05.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 msgr2=0x7fd24814eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 --2- 172.21.15.136:0/439154788 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2480c5040 0x7fd24814eb20 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd24000d5f0 tx=0x7fd24000dab0 comp rx=0 tx=0).stop 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd24eb7e640 1 -- 172.21.15.136:0/439154788 reap_dead start 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 shutdown_connections 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 >> 172.21.15.136:0/439154788 conn(0x7fd2480cdb40 msgr2=0x7fd248146930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 shutdown_connections 2024-04-25T11:45:05.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:05.943+0000 7fd2505e1640 1 -- 172.21.15.136:0/439154788 wait complete. 2024-04-25T11:45:05.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T11:45:06.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 Processor -- start 2024-04-25T11:45:06.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 -- start start 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f965c156750 0x7f965c0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f965c0cfba0 0x7f965c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965c15c410 con 0x7f965c0ca7d0 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f965c15c6b0 con 0x7f965c156750 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f966152c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f965c15c990 con 0x7f965c0cfba0 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f965c156750 0x7f965c0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f965c156750 0x7f965c0ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44610/0 (socket says 172.21.15.136:44610) 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 -- 172.21.15.136:0/771112416 learned_addr learned my addr 172.21.15.136:0/771112416 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965b7fe640 1 --2- 172.21.15.136:0/771112416 >> v2:172.21.15.157:3300/0 conn(0x7f965c0cfba0 0x7f965c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 -- 172.21.15.136:0/771112416 >> v2:172.21.15.157:3300/0 conn(0x7f965c0cfba0 msgr2=0x7f965c06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 --2- 172.21.15.136:0/771112416 >> v2:172.21.15.157:3300/0 conn(0x7f965c0cfba0 0x7f965c06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 -- 172.21.15.136:0/771112416 >> v1:172.21.15.157:6789/0 conn(0x7f965c0ca7d0 legacy=0x7f965c0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.103+0000 7f965affd640 1 -- 172.21.15.136:0/771112416 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c15cc70 con 0x7f965c156750 2024-04-25T11:45:06.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f965affd640 1 --2- 172.21.15.136:0/771112416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c156750 0x7f965c0ca290 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b5850 tx=0x7f964c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0e74828bd08c94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f9659ffb640 1 -- 172.21.15.136:0/771112416 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c0b0070 con 0x7f965c156750 2024-04-25T11:45:06.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f9659ffb640 1 -- 172.21.15.136:0/771112416 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f964c0ab600 con 0x7f965c156750 2024-04-25T11:45:06.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f9659ffb640 1 -- 172.21.15.136:0/771112416 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c0ab9d0 con 0x7f965c156750 2024-04-25T11:45:06.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 -- 172.21.15.136:0/771112416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c156750 msgr2=0x7f965c0ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 --2- 172.21.15.136:0/771112416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c156750 0x7f965c0ca290 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f965c0b5850 tx=0x7f964c01c4e0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 -- 172.21.15.136:0/771112416 shutdown_connections 2024-04-25T11:45:06.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 --2- 172.21.15.136:0/771112416 >> v2:172.21.15.157:3300/0 conn(0x7f965c0cfba0 0x7f965c06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 --2- 172.21.15.136:0/771112416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c156750 0x7f965c0ca290 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 -- 172.21.15.136:0/771112416 >> 172.21.15.136:0/771112416 conn(0x7f965c070f70 msgr2=0x7f965c068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 -- 172.21.15.136:0/771112416 shutdown_connections 2024-04-25T11:45:06.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.107+0000 7f966152c640 1 -- 172.21.15.136:0/771112416 wait complete. 2024-04-25T11:45:06.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 Processor -- start 2024-04-25T11:45:06.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- start start 2024-04-25T11:45:06.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f965c0ca7d0 0x7f965c0ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 0x7f965c14de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1692c0 con 0x7f965c0b4df0 2024-04-25T11:45:06.097 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f965c06da70 con 0x7f965c0ca7d0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f965c169440 con 0x7f965c0cfba0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44618/0 (socket says 172.21.15.136:44618) 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 -- 172.21.15.136:0/2464886186 learned_addr learned my addr 172.21.15.136:0/2464886186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965a7fc640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 0x7f965c14de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965affd640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f965c0ca7d0 0x7f965c0ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 -- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 msgr2=0x7f965c14de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 0x7f965c14de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 -- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f965c0ca7d0 msgr2=0x7f965c0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f965c0ca7d0 0x7f965c0ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 -- 172.21.15.136:0/2464886186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c0ceb80 con 0x7f965c0b4df0 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965a7fc640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 0x7f965c14de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:06.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f965b7fe640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f965400cc80 tx=0x7f965400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540039a0 con 0x7f965c0b4df0 2024-04-25T11:45:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9654003b60 con 0x7f965c0b4df0 2024-04-25T11:45:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- 172.21.15.136:0/2464886186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c0cf320 con 0x7f965c0b4df0 2024-04-25T11:45:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654004020 con 0x7f965c0b4df0 2024-04-25T11:45:06.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f966152c640 1 -- 172.21.15.136:0/2464886186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c0ced90 con 0x7f965c0b4df0 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.111+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96340052a0 con 0x7f965c0b4df0 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.115+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f96540041c0 con 0x7f965c0b4df0 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.115+0000 7f96537fe640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 0x7f963c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.115+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f9654090330 con 0x7f965c0b4df0 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.115+0000 7f965affd640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 0x7f963c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.115+0000 7f965affd640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 0x7f963c071940 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f964c0ad000 tx=0x7f964c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.119+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9654014030 con 0x7f965c0b4df0 2024-04-25T11:45:06.259 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.275+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f9634000bd0 con 0x7f963c06f430 2024-04-25T11:45:06.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.275+0000 7f96537fe640 1 -- 172.21.15.136:0/2464886186 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9634000bd0 con 0x7f963c06f430 2024-04-25T11:45:06.261 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T11:45:06.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 msgr2=0x7f963c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 0x7f963c071940 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f964c0ad000 tx=0x7f964c008040 comp rx=0 tx=0).stop 2024-04-25T11:45:06.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 msgr2=0x7f965c14e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f965400cc80 tx=0x7f965400a5c0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.263 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 shutdown_connections 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f965c0b4df0 0x7f965c14e360 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f963c06f430 0x7f963c071940 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f965c0cfba0 0x7f965c14de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 --2- 172.21.15.136:0/2464886186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f965c0ca7d0 0x7f965c0ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 >> 172.21.15.136:0/2464886186 conn(0x7f965c070f70 msgr2=0x7f965c15ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 shutdown_connections 2024-04-25T11:45:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.279+0000 7f96517fa640 1 -- 172.21.15.136:0/2464886186 wait complete. 2024-04-25T11:45:06.282 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T11:45:06.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:06.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 Processor -- start 2024-04-25T11:45:06.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 -- start start 2024-04-25T11:45:06.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 0x7f14bc142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14bc074170 con 0x7f14bc0743c0 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f14bc074ef0 con 0x7f14bc074ae0 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c3db5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc073e90 con 0x7f14bc0c7900 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c1350640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 0x7f14bc142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.136:53586/0 (socket says 172.21.15.136:53586) 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 -- 172.21.15.136:0/417488614 learned_addr learned my addr 172.21.15.136:0/417488614 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c1350640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 0x7f14bc142920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:39230/0 (socket says 172.21.15.136:39230) 2024-04-25T11:45:06.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c1b51640 1 --1- 172.21.15.136:0/417488614 >> v1:172.21.15.157:6789/0 conn(0x7f14bc0743c0 0x7f14bc159d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:41176/0 (socket says 172.21.15.136:41176) 2024-04-25T11:45:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 -- 172.21.15.136:0/417488614 >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 msgr2=0x7f14bc142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 --2- 172.21.15.136:0/417488614 >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 0x7f14bc142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 -- 172.21.15.136:0/417488614 >> v1:172.21.15.157:6789/0 conn(0x7f14bc0743c0 legacy=0x7f14bc159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.443+0000 7f14c2352640 1 -- 172.21.15.136:0/417488614 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc15d3d0 con 0x7f14bc0c7900 2024-04-25T11:45:06.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c2352640 1 --2- 172.21.15.136:0/417488614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009e50 tx=0x7f14b401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=751016941236031a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c0b4f640 1 -- 172.21.15.136:0/417488614 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40a5700 con 0x7f14bc0c7900 2024-04-25T11:45:06.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c0b4f640 1 -- 172.21.15.136:0/417488614 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b40a58a0 con 0x7f14bc0c7900 2024-04-25T11:45:06.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c0b4f640 1 -- 172.21.15.136:0/417488614 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40a5c70 con 0x7f14bc0c7900 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 -- 172.21.15.136:0/417488614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 msgr2=0x7f14bc0c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 --2- 172.21.15.136:0/417488614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009e50 tx=0x7f14b401ca30 comp rx=0 tx=0).stop 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 -- 172.21.15.136:0/417488614 shutdown_connections 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 --2- 172.21.15.136:0/417488614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc0c7900 0x7f14bc0c7d10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 --2- 172.21.15.136:0/417488614 >> v2:172.21.15.157:3300/0 conn(0x7f14bc074ae0 0x7f14bc142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 -- 172.21.15.136:0/417488614 >> 172.21.15.136:0/417488614 conn(0x7f14bc06de10 msgr2=0x7f14bc0ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 -- 172.21.15.136:0/417488614 shutdown_connections 2024-04-25T11:45:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.447+0000 7f14c3db5640 1 -- 172.21.15.136:0/417488614 wait complete. 2024-04-25T11:45:06.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 Processor -- start 2024-04-25T11:45:06.435 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- start start 2024-04-25T11:45:06.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14bc0743c0 0x7f14bc164f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc074ae0 0x7f14bc0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 0x7f14bc0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc15a450 con 0x7f14bc0c7900 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc15e190 con 0x7f14bc0743c0 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc0c4780 con 0x7f14bc074ae0 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 0x7f14bc0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 0x7f14bc0c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44634/0 (socket says 172.21.15.136:44634) 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 -- 172.21.15.136:0/1871228499 learned_addr learned my addr 172.21.15.136:0/1871228499 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c1b51640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14bc0743c0 0x7f14bc164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c1350640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc074ae0 0x7f14bc0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 -- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc074ae0 msgr2=0x7f14bc0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc074ae0 0x7f14bc0ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 -- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14bc0743c0 msgr2=0x7f14bc164f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14bc0743c0 0x7f14bc164f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc0c5e70 con 0x7f14bc0c7900 2024-04-25T11:45:06.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c1b51640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14bc0743c0 0x7f14bc164f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c1350640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14bc074ae0 0x7f14bc0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c2352640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 0x7f14bc0c6c30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f14b401a660 tx=0x7f14b40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40bc030 con 0x7f14bc0c7900 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b40b8470 con 0x7f14bc0c7900 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc0cd980 con 0x7f14bc0c7900 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40b8880 con 0x7f14bc0c7900 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.451+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc0cdf40 con 0x7f14bc0c7900 2024-04-25T11:45:06.440 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14bc06a3c0 con 0x7f14bc0c7900 2024-04-25T11:45:06.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f14b40bf070 con 0x7f14bc0c7900 2024-04-25T11:45:06.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f14a006f400 0x7f14a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.441 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f14b412a1e0 con 0x7f14bc0c7900 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7f14a0079880 0x7f14a007bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14a007c440 con 0x7f14a0079880 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f14b4131050 con 0x7f14bc0c7900 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14c1b51640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f14a006f400 0x7f14a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14c1350640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7f14a0079880 0x7f14a007bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14c1350640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7f14a0079880 0x7f14a007bd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14c1b51640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f14a006f400 0x7f14a0071910 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f70 tx=0x7f14b8007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.455+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== osd.1 v2:172.21.15.136:6824/1933356212 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14a007c440 con 0x7f14a0079880 2024-04-25T11:45:06.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.475+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 --> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f14bc159080 con 0x7f14a0079880 2024-04-25T11:45:06.462 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14b27fc640 1 -- 172.21.15.136:0/1871228499 <== osd.1 v2:172.21.15.136:6824/1933356212 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f14bc159080 con 0x7f14a0079880 2024-04-25T11:45:06.463 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:06.463 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:06.463 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7f14a0079880 msgr2=0x7f14a007bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:6824/1933356212,v1:172.21.15.136:6825/1933356212] conn(0x7f14a0079880 0x7f14a007bd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f14a006f400 msgr2=0x7f14a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f14a006f400 0x7f14a0071910 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000f70 tx=0x7f14b8007d70 comp rx=0 tx=0).stop 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 msgr2=0x7f14bc0c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 --2- 172.21.15.136:0/1871228499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14bc0c7900 0x7f14bc0c6c30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f14b401a660 tx=0x7f14b40197d0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c2352640 1 -- 172.21.15.136:0/1871228499 reap_dead start 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 shutdown_connections 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 >> 172.21.15.136:0/1871228499 conn(0x7f14bc06de10 msgr2=0x7f14bc0c4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 shutdown_connections 2024-04-25T11:45:06.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.479+0000 7f14c3db5640 1 -- 172.21.15.136:0/1871228499 wait complete. 2024-04-25T11:45:06.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T11:45:06.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.631+0000 7feb710bb640 1 Processor -- start 2024-04-25T11:45:06.617 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.631+0000 7feb710bb640 1 -- start start 2024-04-25T11:45:06.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.631+0000 7feb710bb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feb6c0c96e0 0x7feb6c1540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb710bb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feb6c15aae0 0x7feb6c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb710bb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c0c9af0 con 0x7feb6c0c79e0 2024-04-25T11:45:06.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb710bb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7feb6c06cc10 con 0x7feb6c15aae0 2024-04-25T11:45:06.619 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb710bb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7feb6c0cb220 con 0x7feb6c0c96e0 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feb6c15aae0 0x7feb6c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7feb6c0c79e0 0x7feb6c15a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46482/0 (socket says 172.21.15.136:46482) 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6a59c640 1 -- 172.21.15.136:0/3464987210 learned_addr learned my addr 172.21.15.136:0/3464987210 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6ad9d640 1 --2- 172.21.15.136:0/3464987210 >> v2:172.21.15.157:3300/0 conn(0x7feb6c0c96e0 0x7feb6c1540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 -- 172.21.15.136:0/3464987210 >> v2:172.21.15.157:3300/0 conn(0x7feb6c0c96e0 msgr2=0x7feb6c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 --2- 172.21.15.136:0/3464987210 >> v2:172.21.15.157:3300/0 conn(0x7feb6c0c96e0 0x7feb6c1540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 -- 172.21.15.136:0/3464987210 >> v1:172.21.15.136:6789/0 conn(0x7feb6c0c79e0 legacy=0x7feb6c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb69d9b640 1 -- 172.21.15.136:0/3464987210 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084176768 0 0) 0x7feb6c0c9af0 con 0x7feb6c0c79e0 2024-04-25T11:45:06.620 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 -- 172.21.15.136:0/3464987210 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c0cb500 con 0x7feb6c15aae0 2024-04-25T11:45:06.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb6b59e640 1 --2- 172.21.15.136:0/3464987210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c0c7df0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feb6c070790 tx=0x7feb6401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eabd9960e05ac97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb69d9b640 1 -- 172.21.15.136:0/3464987210 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640b4070 con 0x7feb6c15aae0 2024-04-25T11:45:06.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb69d9b640 1 -- 172.21.15.136:0/3464987210 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb640af600 con 0x7feb6c15aae0 2024-04-25T11:45:06.622 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.635+0000 7feb69d9b640 1 -- 172.21.15.136:0/3464987210 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640af9d0 con 0x7feb6c15aae0 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- 172.21.15.136:0/3464987210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 msgr2=0x7feb6c0c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- 172.21.15.136:0/3464987210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c0c7df0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7feb6c070790 tx=0x7feb6401f110 comp rx=0 tx=0).stop 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- 172.21.15.136:0/3464987210 shutdown_connections 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- 172.21.15.136:0/3464987210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c0c7df0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- 172.21.15.136:0/3464987210 >> v2:172.21.15.157:3300/0 conn(0x7feb6c0c96e0 0x7feb6c1540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.623 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- 172.21.15.136:0/3464987210 >> 172.21.15.136:0/3464987210 conn(0x7feb6c06f9d0 msgr2=0x7feb6c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- 172.21.15.136:0/3464987210 shutdown_connections 2024-04-25T11:45:06.624 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- 172.21.15.136:0/3464987210 wait complete. 2024-04-25T11:45:06.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 Processor -- start 2024-04-25T11:45:06.625 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- start start 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb6c0c79e0 0x7feb6c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 0x7feb6c14f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c0cf0f0 con 0x7feb6c15aae0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c154610 con 0x7feb6c0c96e0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb710bb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feb6c0ced40 con 0x7feb6c0c79e0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb6b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb6b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44662/0 (socket says 172.21.15.136:44662) 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.639+0000 7feb6b59e640 1 -- 172.21.15.136:0/1508625769 learned_addr learned my addr 172.21.15.136:0/1508625769 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6ad9d640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb6c0c79e0 0x7feb6c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6a59c640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 0x7feb6c14f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.628 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 -- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb6c0c79e0 msgr2=0x7feb6c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb6c0c79e0 0x7feb6c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 -- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 msgr2=0x7feb6c14f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 0x7feb6c14f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 -- 172.21.15.136:0/1508625769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c1539a0 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6a59c640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 0x7feb6c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6b59e640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7feb640b2040 tx=0x7feb6401beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640b4040 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb640c1070 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c0c7810 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64009450 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb6c150750 con 0x7feb6c15aae0 2024-04-25T11:45:06.629 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6c15b680 con 0x7feb6c15aae0 2024-04-25T11:45:06.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feb640ba6b0 con 0x7feb6c15aae0 2024-04-25T11:45:06.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 0x7feb58071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7feb6412ef30 con 0x7feb6c15aae0 2024-04-25T11:45:06.630 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.643+0000 7feb6ad9d640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 0x7feb58071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.631 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.647+0000 7feb6ad9d640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 0x7feb58071940 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b930 tx=0x7feb60008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.633 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.647+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feb64018c70 con 0x7feb6c15aae0 2024-04-25T11:45:06.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.803+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7feb6c06a350 con 0x7feb5806f430 2024-04-25T11:45:06.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb537fe640 1 -- 172.21.15.136:0/1508625769 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7feb6c06a350 con 0x7feb5806f430 2024-04-25T11:45:06.791 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 msgr2=0x7feb58071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 0x7feb58071940 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feb6000b930 tx=0x7feb60008040 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 msgr2=0x7feb6c14fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7feb640b2040 tx=0x7feb6401beb0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 shutdown_connections 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb6c15aae0 0x7feb6c14fc10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb6c0c96e0 0x7feb6c14f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feb5806f430 0x7feb58071940 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 --2- 172.21.15.136:0/1508625769 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb6c0c79e0 0x7feb6c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 >> 172.21.15.136:0/1508625769 conn(0x7feb6c06f9d0 msgr2=0x7feb6c075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 shutdown_connections 2024-04-25T11:45:06.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.807+0000 7feb710bb640 1 -- 172.21.15.136:0/1508625769 wait complete. 2024-04-25T11:45:06.811 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T11:45:06.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:06.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 Processor -- start 2024-04-25T11:45:06.968 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 -- start start 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9900cbf50 0x7fc9900c7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc9900cc360 0x7fc9900c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc99015db00 0x7fc99015df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fc99006dca0 con 0x7fc9900cc360 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc99015d670 con 0x7fc99015db00 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.983+0000 7fc9965ea640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9900cbc60 con 0x7fc9900cbf50 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc9900cc360 0x7fc9900c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc9900cc360 0x7fc9900c7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44670/0 (socket says 172.21.15.136:44670) 2024-04-25T11:45:06.971 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 -- 172.21.15.136:0/117001416 learned_addr learned my addr 172.21.15.136:0/117001416 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98ffff640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9900cbf50 0x7fc9900c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc994b87640 1 --2- 172.21.15.136:0/117001416 >> v2:172.21.15.157:3300/0 conn(0x7fc99015db00 0x7fc99015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 -- 172.21.15.136:0/117001416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9900cbf50 msgr2=0x7fc9900c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9900cbf50 0x7fc9900c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 -- 172.21.15.136:0/117001416 >> v2:172.21.15.157:3300/0 conn(0x7fc99015db00 msgr2=0x7fc99015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/117001416 >> v2:172.21.15.157:3300/0 conn(0x7fc99015db00 0x7fc99015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.972 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 -- 172.21.15.136:0/117001416 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9900c8540 con 0x7fc9900cc360 2024-04-25T11:45:06.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900c7790 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc9900cebc0 tx=0x7fc98401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da56f3c19be22c88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98effd640 1 -- 172.21.15.136:0/117001416 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840b0070 con 0x7fc9900cc360 2024-04-25T11:45:06.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98effd640 1 -- 172.21.15.136:0/117001416 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9840ab600 con 0x7fc9900cc360 2024-04-25T11:45:06.973 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc98effd640 1 -- 172.21.15.136:0/117001416 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840ab9d0 con 0x7fc9900cc360 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 -- 172.21.15.136:0/117001416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 msgr2=0x7fc9900c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900c7790 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc9900cebc0 tx=0x7fc98401c490 comp rx=0 tx=0).stop 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 -- 172.21.15.136:0/117001416 shutdown_connections 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900c7790 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 --2- 172.21.15.136:0/117001416 >> v2:172.21.15.157:3300/0 conn(0x7fc99015db00 0x7fc99015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 --2- 172.21.15.136:0/117001416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9900cbf50 0x7fc9900c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.974 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 -- 172.21.15.136:0/117001416 >> 172.21.15.136:0/117001416 conn(0x7fc9900cdb40 msgr2=0x7fc9900cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 -- 172.21.15.136:0/117001416 shutdown_connections 2024-04-25T11:45:06.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.987+0000 7fc9965ea640 1 -- 172.21.15.136:0/117001416 wait complete. 2024-04-25T11:45:06.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 Processor -- start 2024-04-25T11:45:06.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- start start 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9900cbf50 0x7fc990139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900734d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99015db00 0x7fc990074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc9900c4b50 con 0x7fc9900cc360 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9900c90c0 con 0x7fc9900cbf50 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9900c4cd0 con 0x7fc99015db00 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.978 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900734d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44676/0 (socket says 172.21.15.136:44676) 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 -- 172.21.15.136:0/215442396 learned_addr learned my addr 172.21.15.136:0/215442396 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98ffff640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9900cbf50 0x7fc990139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc994b87640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99015db00 0x7fc990074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 -- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99015db00 msgr2=0x7fc990074a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99015db00 0x7fc990074a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 -- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9900cbf50 msgr2=0x7fc990139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9900cbf50 0x7fc990139560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc990157dd0 con 0x7fc9900cc360 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc994b87640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc99015db00 0x7fc990074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98f7fe640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900734d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc98401ab40 tx=0x7fc984019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840ba070 con 0x7fc9900cc360 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9840b4620 con 0x7fc9900cc360 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc990137480 con 0x7fc9900cc360 2024-04-25T11:45:06.979 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840b0040 con 0x7fc9900cc360 2024-04-25T11:45:06.980 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.991+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc990136440 con 0x7fc9900cc360 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc99006a3c0 con 0x7fc9900cc360 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc9840bd070 con 0x7fc9900cc360 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc97c06f400 0x7fc97c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fc984128cf0 con 0x7fc9900cc360 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc97c079880 0x7fc97c07bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc984007e50 con 0x7fc97c079880 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc98412b3f0 con 0x7fc9900cc360 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98ffff640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc97c06f400 0x7fc97c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.981 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc994b87640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc97c079880 0x7fc97c07bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:06.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc994b87640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc97c079880 0x7fc97c07bd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98ffff640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc97c06f400 0x7fc97c071910 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc988004470 tx=0x7fc988003a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:06.982 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:06.995+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== osd.2 v2:172.21.15.136:6816/4197312530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc984007e50 con 0x7fc97c079880 2024-04-25T11:45:06.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.007+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 --> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc99013acc0 con 0x7fc97c079880 2024-04-25T11:45:06.994 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc98cff9640 1 -- 172.21.15.136:0/215442396 <== osd.2 v2:172.21.15.136:6816/4197312530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc980009410 con 0x7fc97c079880 2024-04-25T11:45:06.995 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:06.995 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:06.995 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc97c079880 msgr2=0x7fc97c07bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:6816/4197312530,v1:172.21.15.136:6817/4197312530] conn(0x7fc97c079880 0x7fc97c07bd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc97c06f400 msgr2=0x7fc97c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc97c06f400 0x7fc97c071910 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc988004470 tx=0x7fc988003a40 comp rx=0 tx=0).stop 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 msgr2=0x7fc9900734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 --2- 172.21.15.136:0/215442396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9900cc360 0x7fc9900734d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc98401ab40 tx=0x7fc984019230 comp rx=0 tx=0).stop 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc994b87640 1 -- 172.21.15.136:0/215442396 reap_dead start 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 shutdown_connections 2024-04-25T11:45:06.996 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 >> 172.21.15.136:0/215442396 conn(0x7fc9900cdb40 msgr2=0x7fc990152e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:06.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 shutdown_connections 2024-04-25T11:45:06.997 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.011+0000 7fc9965ea640 1 -- 172.21.15.136:0/215442396 wait complete. 2024-04-25T11:45:07.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T11:45:07.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.163+0000 7ff1556a4640 1 Processor -- start 2024-04-25T11:45:07.149 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.163+0000 7ff1556a4640 1 -- start start 2024-04-25T11:45:07.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1500c96e0 0x7ff1501540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.151 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1500c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff15015a3d0 0x7ff15015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff15006cc10 con 0x7ff1500c96e0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff1500c9af0 con 0x7ff15015a3d0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1500cb220 con 0x7ff1500c79e0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1500c96e0 0x7ff1501540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1500c96e0 0x7ff1501540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44688/0 (socket says 172.21.15.136:44688) 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 -- 172.21.15.136:0/499880696 learned_addr learned my addr 172.21.15.136:0/499880696 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14f7fe640 1 --2- 172.21.15.136:0/499880696 >> v2:172.21.15.157:3300/0 conn(0x7ff15015a3d0 0x7ff15015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14e7fc640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1500c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 -- 172.21.15.136:0/499880696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 msgr2=0x7ff1500c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1500c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 -- 172.21.15.136:0/499880696 >> v2:172.21.15.157:3300/0 conn(0x7ff15015a3d0 msgr2=0x7ff15015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 --2- 172.21.15.136:0/499880696 >> v2:172.21.15.157:3300/0 conn(0x7ff15015a3d0 0x7ff15015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 -- 172.21.15.136:0/499880696 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1500cb500 con 0x7ff1500c96e0 2024-04-25T11:45:07.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14effd640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1500c96e0 0x7ff1501540d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff1500729b0 tx=0x7ff14001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe751c4b7b58a778 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14dffb640 1 -- 172.21.15.136:0/499880696 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400b5070 con 0x7ff1500c96e0 2024-04-25T11:45:07.153 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14dffb640 1 -- 172.21.15.136:0/499880696 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1400a89d0 con 0x7ff1500c96e0 2024-04-25T11:45:07.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff14dffb640 1 -- 172.21.15.136:0/499880696 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1400a8da0 con 0x7ff1500c96e0 2024-04-25T11:45:07.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- 172.21.15.136:0/499880696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1500c96e0 msgr2=0x7ff1501540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1500c96e0 0x7ff1501540d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff1500729b0 tx=0x7ff14001ed10 comp rx=0 tx=0).stop 2024-04-25T11:45:07.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- 172.21.15.136:0/499880696 shutdown_connections 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- 172.21.15.136:0/499880696 >> v2:172.21.15.157:3300/0 conn(0x7ff15015a3d0 0x7ff15015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1500c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 --2- 172.21.15.136:0/499880696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1500c96e0 0x7ff1501540d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- 172.21.15.136:0/499880696 >> 172.21.15.136:0/499880696 conn(0x7ff15006f9d0 msgr2=0x7ff15006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- 172.21.15.136:0/499880696 shutdown_connections 2024-04-25T11:45:07.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.167+0000 7ff1556a4640 1 -- 172.21.15.136:0/499880696 wait complete. 2024-04-25T11:45:07.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 Processor -- start 2024-04-25T11:45:07.156 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 -- start start 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1501e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1500c96e0 0x7ff150156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff150154610 con 0x7ff15015a3d0 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff15015ad20 con 0x7ff1500c96e0 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1556a4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1500cf0f0 con 0x7ff1500c79e0 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44694/0 (socket says 172.21.15.136:44694) 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 -- 172.21.15.136:0/2251634285 learned_addr learned my addr 172.21.15.136:0/2251634285 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:07.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14e7fc640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1500c96e0 0x7ff150156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14effd640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1501e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 -- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 msgr2=0x7ff1501e3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1501e3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 -- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1500c96e0 msgr2=0x7ff150156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1500c96e0 0x7ff150156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 -- 172.21.15.136:0/2251634285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1501e3470 con 0x7ff15015a3d0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff14f7fe640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff148007d40 tx=0x7ff14800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148014420 con 0x7ff15015a3d0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff148019070 con 0x7ff15015a3d0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.171+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148014820 con 0x7ff15015a3d0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1501e4140 con 0x7ff15015a3d0 2024-04-25T11:45:07.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1501e5d30 con 0x7ff15015a3d0 2024-04-25T11:45:07.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15006a350 con 0x7ff15015a3d0 2024-04-25T11:45:07.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1480149c0 con 0x7ff15015a3d0 2024-04-25T11:45:07.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1377fe640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 0x7ff13c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7ff148090370 con 0x7ff15015a3d0 2024-04-25T11:45:07.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff14effd640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 0x7ff13c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.175+0000 7ff14effd640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 0x7ff13c071910 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff1400b2000 tx=0x7ff140008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.179+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff14805ca30 con 0x7ff15015a3d0 2024-04-25T11:45:07.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.335+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ff150147640 con 0x7ff13c06f400 2024-04-25T11:45:07.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.335+0000 7ff1377fe640 1 -- 172.21.15.136:0/2251634285 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff150147640 con 0x7ff13c06f400 2024-04-25T11:45:07.320 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 msgr2=0x7ff13c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 0x7ff13c071910 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff1400b2000 tx=0x7ff140008040 comp rx=0 tx=0).stop 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 msgr2=0x7ff150154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff148007d40 tx=0x7ff14800e4b0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 shutdown_connections 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff15015a3d0 0x7ff150154a40 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1500c96e0 0x7ff150156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff13c06f400 0x7ff13c071910 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 --2- 172.21.15.136:0/2251634285 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1500c79e0 0x7ff1501e3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 >> 172.21.15.136:0/2251634285 conn(0x7ff15006f9d0 msgr2=0x7ff150148cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 shutdown_connections 2024-04-25T11:45:07.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.339+0000 7ff1556a4640 1 -- 172.21.15.136:0/2251634285 wait complete. 2024-04-25T11:45:07.346 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T11:45:07.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:07.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 Processor -- start 2024-04-25T11:45:07.491 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 -- start start 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f32ac0741c0 0x7f32ac15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0c2e00 0x7f32ac0c3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32ac15c1a0 con 0x7f32ac0c4460 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f32ac0745d0 con 0x7f32ac0741c0 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32b1e2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac0c4870 con 0x7f32ac0c2e00 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f32ac0741c0 0x7f32ac15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f32ac0741c0 0x7f32ac15cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44696/0 (socket says 172.21.15.136:44696) 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 -- 172.21.15.136:0/1861857642 learned_addr learned my addr 172.21.15.136:0/1861857642 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:07.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32aaffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f32ac0c4460 0x7f32ac0c89c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46492/0 (socket says 172.21.15.136:46492) 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32abfff640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0c2e00 0x7f32ac0c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 -- 172.21.15.136:0/1861857642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0c2e00 msgr2=0x7f32ac0c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0c2e00 0x7f32ac0c3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 -- 172.21.15.136:0/1861857642 >> v1:172.21.15.136:6789/0 conn(0x7f32ac0c4460 legacy=0x7f32ac0c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32ab7fe640 1 -- 172.21.15.136:0/1861857642 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac0c9d80 con 0x7f32ac0741c0 2024-04-25T11:45:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.507+0000 7f32aa7fc640 1 -- 172.21.15.136:0/1861857642 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228100403 0 0) 0x7f32ac15c1a0 con 0x7f32ac0c4460 2024-04-25T11:45:07.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32ab7fe640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0741c0 0x7f32ac15cc40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009e50 tx=0x7f32a401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35e464d973dc6ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32aa7fc640 1 -- 172.21.15.136:0/1861857642 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a40a5660 con 0x7f32ac0741c0 2024-04-25T11:45:07.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32aa7fc640 1 -- 172.21.15.136:0/1861857642 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a40a5800 con 0x7f32ac0741c0 2024-04-25T11:45:07.496 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32aa7fc640 1 -- 172.21.15.136:0/1861857642 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a40a5bd0 con 0x7f32ac0741c0 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 -- 172.21.15.136:0/1861857642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0741c0 msgr2=0x7f32ac15cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0741c0 0x7f32ac15cc40 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009e50 tx=0x7f32a401c9a0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 -- 172.21.15.136:0/1861857642 shutdown_connections 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0c2e00 0x7f32ac0c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/1861857642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0741c0 0x7f32ac15cc40 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 -- 172.21.15.136:0/1861857642 >> 172.21.15.136:0/1861857642 conn(0x7f32ac0ca5a0 msgr2=0x7f32ac0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 -- 172.21.15.136:0/1861857642 shutdown_connections 2024-04-25T11:45:07.497 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.511+0000 7f32b1e2f640 1 -- 172.21.15.136:0/1861857642 wait complete. 2024-04-25T11:45:07.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 Processor -- start 2024-04-25T11:45:07.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- start start 2024-04-25T11:45:07.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0741c0 0x7f32ac148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32ac0c2e00 0x7f32ac147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 0x7f32ac06c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac0ca320 con 0x7f32ac0c4460 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32ac15c720 con 0x7f32ac0c2e00 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32ac169230 con 0x7f32ac0741c0 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 0x7f32ac06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 0x7f32ac06c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44698/0 (socket says 172.21.15.136:44698) 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 -- 172.21.15.136:0/840883744 learned_addr learned my addr 172.21.15.136:0/840883744 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32ab7fe640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0741c0 0x7f32ac148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32aaffd640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32ac0c2e00 0x7f32ac147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 -- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0741c0 msgr2=0x7f32ac148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.502 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32ac0741c0 0x7f32ac148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 -- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32ac0c2e00 msgr2=0x7f32ac147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32ac0c2e00 0x7f32ac147620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32ac1489a0 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32abfff640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 0x7f32ac06c670 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f329c00d550 tx=0x7f329c00da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c0125f0 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329c019070 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ac153f00 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c0129d0 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ac154f00 con 0x7f32ac0c4460 2024-04-25T11:45:07.503 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.515+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f32ac06a3c0 con 0x7f32ac0c4460 2024-04-25T11:45:07.504 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f329c012b70 con 0x7f32ac0c4460 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f328c06f430 0x7f328c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f329c090850 con 0x7f32ac0c4460 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f328c0798b0 0x7f328c07bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f329c004480 con 0x7f328c0798b0 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f329c090ce0 con 0x7f32ac0c4460 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32ab7fe640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f328c06f430 0x7f328c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32aaffd640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f328c0798b0 0x7f328c07bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32aaffd640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f328c0798b0 0x7f328c07bda0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32ab7fe640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f328c06f430 0x7f328c071940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f32a40ae000 tx=0x7f32a40a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.505 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.519+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== osd.3 v2:172.21.15.136:6808/2427382821 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f329c004480 con 0x7f328c0798b0 2024-04-25T11:45:07.517 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 --> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f32ac0c6150 con 0x7f328c0798b0 2024-04-25T11:45:07.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32a3fff640 1 -- 172.21.15.136:0/840883744 <== osd.3 v2:172.21.15.136:6808/2427382821 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f32ac0c6150 con 0x7f328c0798b0 2024-04-25T11:45:07.518 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:07.518 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:07.518 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f328c0798b0 msgr2=0x7f328c07bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:6808/2427382821,v1:172.21.15.136:6809/2427382821] conn(0x7f328c0798b0 0x7f328c07bda0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f328c06f430 msgr2=0x7f328c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f328c06f430 0x7f328c071940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f32a40ae000 tx=0x7f32a40a7070 comp rx=0 tx=0).stop 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 msgr2=0x7f32ac06c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 --2- 172.21.15.136:0/840883744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32ac0c4460 0x7f32ac06c670 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f329c00d550 tx=0x7f329c00da10 comp rx=0 tx=0).stop 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32abfff640 1 -- 172.21.15.136:0/840883744 reap_dead start 2024-04-25T11:45:07.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 shutdown_connections 2024-04-25T11:45:07.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 >> 172.21.15.136:0/840883744 conn(0x7f32ac0ca5a0 msgr2=0x7f32ac068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 shutdown_connections 2024-04-25T11:45:07.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.531+0000 7f32b1e2f640 1 -- 172.21.15.136:0/840883744 wait complete. 2024-04-25T11:45:07.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T11:45:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 Processor -- start 2024-04-25T11:45:07.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 -- start start 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f844815a430 0x7f844815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f844815d750 con 0x7f84480b5060 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f844815da30 con 0x7f844815a430 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f844e017640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f844815dd10 con 0x7f84480ca290 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f844815a430 0x7f844815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f84477fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f84480b5060 0x7f84480c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46504/0 (socket says 172.21.15.136:46504) 2024-04-25T11:45:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f844815a430 0x7f844815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44704/0 (socket says 172.21.15.136:44704) 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f84477fe640 1 -- 172.21.15.136:0/2294653012 learned_addr learned my addr 172.21.15.136:0/2294653012 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8446ffd640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 -- 172.21.15.136:0/2294653012 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 msgr2=0x7f84480ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 -- 172.21.15.136:0/2294653012 >> v1:172.21.15.136:6789/0 conn(0x7f84480b5060 legacy=0x7f84480c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f8447fff640 1 -- 172.21.15.136:0/2294653012 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844815dff0 con 0x7f844815a430 2024-04-25T11:45:07.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.687+0000 7f84467fc640 1 -- 172.21.15.136:0/2294653012 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577441293 0 0) 0x7f844815d750 con 0x7f84480b5060 2024-04-25T11:45:07.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f8447fff640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844815a840 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f84400097c0 tx=0x7f844001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28923bc20eda38fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f84467fc640 1 -- 172.21.15.136:0/2294653012 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400b5070 con 0x7f844815a430 2024-04-25T11:45:07.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f84467fc640 1 -- 172.21.15.136:0/2294653012 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84400a8990 con 0x7f844815a430 2024-04-25T11:45:07.677 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f84467fc640 1 -- 172.21.15.136:0/2294653012 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400a8d60 con 0x7f844815a430 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 -- 172.21.15.136:0/2294653012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 msgr2=0x7f844815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844815a840 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f84400097c0 tx=0x7f844001ec90 comp rx=0 tx=0).stop 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 -- 172.21.15.136:0/2294653012 shutdown_connections 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844815a840 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 --2- 172.21.15.136:0/2294653012 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 -- 172.21.15.136:0/2294653012 >> 172.21.15.136:0/2294653012 conn(0x7f8448070f70 msgr2=0x7f8448068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 -- 172.21.15.136:0/2294653012 shutdown_connections 2024-04-25T11:45:07.678 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.691+0000 7f844e017640 1 -- 172.21.15.136:0/2294653012 wait complete. 2024-04-25T11:45:07.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 Processor -- start 2024-04-25T11:45:07.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- start start 2024-04-25T11:45:07.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 0x7f8448145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f844815ad80 con 0x7f844815a430 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8448159f10 con 0x7f84480b5060 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f844815d3f0 con 0x7f84480ca290 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44714/0 (socket says 172.21.15.136:44714) 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 -- 172.21.15.136:0/1985201568 learned_addr learned my addr 172.21.15.136:0/1985201568 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8446ffd640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f84477fe640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 0x7f8448145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 -- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 msgr2=0x7f84480c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 -- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 msgr2=0x7f8448145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 0x7f8448145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 -- 172.21.15.136:0/1985201568 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8448146330 con 0x7f844815a430 2024-04-25T11:45:07.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f84477fe640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 0x7f8448145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f8447fff640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f844001c8c0 tx=0x7f844001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400bf070 con 0x7f844815a430 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84400b0780 con 0x7f844815a430 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8448151e50 con 0x7f844815a430 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400b5040 con 0x7f844815a430 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.695+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844814ea60 con 0x7f844815a430 2024-04-25T11:45:07.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f844814e4f0 con 0x7f844815a430 2024-04-25T11:45:07.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f84400c2070 con 0x7f844815a430 2024-04-25T11:45:07.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f842ffff640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 0x7f8434071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:07.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f844012c420 con 0x7f844815a430 2024-04-25T11:45:07.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f84477fe640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 0x7f8434071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:07.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.699+0000 7f84477fe640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 0x7f8434071940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f84380012f0 tx=0x7f8438009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:07.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.707+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f84400187f0 con 0x7f844815a430 2024-04-25T11:45:07.855 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.871+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f844806a350 con 0x7f843406f430 2024-04-25T11:45:07.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.871+0000 7f842ffff640 1 -- 172.21.15.136:0/1985201568 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f844806a350 con 0x7f843406f430 2024-04-25T11:45:07.856 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T11:45:07.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 msgr2=0x7f8434071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 0x7f8434071940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f84380012f0 tx=0x7f8438009040 comp rx=0 tx=0).stop 2024-04-25T11:45:07.859 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 msgr2=0x7f844813dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f844001c8c0 tx=0x7f844001ba30 comp rx=0 tx=0).stop 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 shutdown_connections 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f844815a430 0x7f844813dd20 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f843406f430 0x7f8434071940 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84480ca290 0x7f84480c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 --2- 172.21.15.136:0/1985201568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84480b5060 0x7f8448145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 >> 172.21.15.136:0/1985201568 conn(0x7f8448070f70 msgr2=0x7f84480ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 shutdown_connections 2024-04-25T11:45:07.860 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:07.875+0000 7f844e017640 1 -- 172.21.15.136:0/1985201568 wait complete. 2024-04-25T11:45:07.877 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T11:45:07.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:08.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.047+0000 7feed03c6640 1 Processor -- start 2024-04-25T11:45:08.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.047+0000 7feed03c6640 1 -- start start 2024-04-25T11:45:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 0x7feec80cb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 0x7feec81530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feec80c7aa0 0x7feec80c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7feec8074d90 con 0x7feec80c7aa0 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7feec81535e0 con 0x7feec8152c90 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feed03c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feec8074ad0 con 0x7feec80cb100 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feec80c7aa0 0x7feec80c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feec80c7aa0 0x7feec80c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44728/0 (socket says 172.21.15.136:44728) 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 -- 172.21.15.136:0/1120997118 learned_addr learned my addr 172.21.15.136:0/1120997118 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feecd961640 1 --2- 172.21.15.136:0/1120997118 >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 0x7feec81530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece162640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 0x7feec80cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 -- 172.21.15.136:0/1120997118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 msgr2=0x7feec80cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 0x7feec80cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 -- 172.21.15.136:0/1120997118 >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 msgr2=0x7feec81530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 --2- 172.21.15.136:0/1120997118 >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 0x7feec81530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 -- 172.21.15.136:0/1120997118 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec80c5890 con 0x7feec80c7aa0 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece162640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 0x7feec80cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:08.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feecd961640 1 --2- 172.21.15.136:0/1120997118 >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 0x7feec81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.051+0000 7feece963640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec80c7eb0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7feec8074f70 tx=0x7feec401eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb5f6c07fe02fff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feecd160640 1 -- 172.21.15.136:0/1120997118 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec400b2d0 con 0x7feec80c7aa0 2024-04-25T11:45:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feecd160640 1 -- 172.21.15.136:0/1120997118 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec400b470 con 0x7feec80c7aa0 2024-04-25T11:45:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feecd160640 1 -- 172.21.15.136:0/1120997118 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec40a7a10 con 0x7feec80c7aa0 2024-04-25T11:45:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- 172.21.15.136:0/1120997118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 msgr2=0x7feec80c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec80c7eb0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7feec8074f70 tx=0x7feec401eb60 comp rx=0 tx=0).stop 2024-04-25T11:45:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- 172.21.15.136:0/1120997118 shutdown_connections 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec80c7eb0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 --2- 172.21.15.136:0/1120997118 >> v2:172.21.15.157:3300/0 conn(0x7feec8152c90 0x7feec81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 --2- 172.21.15.136:0/1120997118 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec80cb100 0x7feec80cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- 172.21.15.136:0/1120997118 >> 172.21.15.136:0/1120997118 conn(0x7feec806d950 msgr2=0x7feec806dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- 172.21.15.136:0/1120997118 shutdown_connections 2024-04-25T11:45:08.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- 172.21.15.136:0/1120997118 wait complete. 2024-04-25T11:45:08.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 Processor -- start 2024-04-25T11:45:08.042 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.055+0000 7feed03c6640 1 -- start start 2024-04-25T11:45:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec8153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.045 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec80cb100 0x7feec81f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec8152c90 0x7feec814e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feec80caa60 con 0x7feec80c7aa0 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feec80c83f0 con 0x7feec80cb100 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feec80cabe0 con 0x7feec8152c90 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec8153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec8153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44730/0 (socket says 172.21.15.136:44730) 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 -- 172.21.15.136:0/807864246 learned_addr learned my addr 172.21.15.136:0/807864246 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece963640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec8152c90 0x7feec814e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feecd961640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec80cb100 0x7feec81f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 -- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec8152c90 msgr2=0x7feec814e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feec8152c90 0x7feec814e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 -- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec80cb100 msgr2=0x7feec81f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec80cb100 0x7feec81f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec81f4930 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feecd961640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feec80cb100 0x7feec81f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feece162640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec8153790 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7feec000c7f0 tx=0x7feec000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0015070 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feec0007820 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec8155920 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec0007c10 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec8155ee0 con 0x7feec80c7aa0 2024-04-25T11:45:08.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.059+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feec81541d0 con 0x7feec80c7aa0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feec0018070 con 0x7feec80c7aa0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feeac06f430 0x7feeac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7feec0090220 con 0x7feec80c7aa0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7feeac0798b0 0x7feeac07bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feeac07c470 con 0x7feeac0798b0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7feec00906b0 con 0x7feec80c7aa0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feecd961640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feeac06f430 0x7feeac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feece963640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7feeac0798b0 0x7feeac07bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feecd961640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feeac06f430 0x7feeac071940 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7feec8075d40 tx=0x7feeb8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.063+0000 7feece963640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7feeac0798b0 0x7feeac07bda0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.067+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== osd.4 v2:172.21.15.157:6808/3530978019 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feeac07c470 con 0x7feeac0798b0 2024-04-25T11:45:08.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 --> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7feec806a3c0 con 0x7feeac0798b0 2024-04-25T11:45:08.064 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feebeffd640 1 -- 172.21.15.136:0/807864246 <== osd.4 v2:172.21.15.157:6808/3530978019 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7feec806a3c0 con 0x7feeac0798b0 2024-04-25T11:45:08.064 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:08.064 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:08.064 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7feeac0798b0 msgr2=0x7feeac07bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6808/3530978019,v1:172.21.15.157:6810/3530978019] conn(0x7feeac0798b0 0x7feeac07bda0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feeac06f430 msgr2=0x7feeac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7feeac06f430 0x7feeac071940 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7feec8075d40 tx=0x7feeb8008040 comp rx=0 tx=0).stop 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 msgr2=0x7feec8153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 --2- 172.21.15.136:0/807864246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feec80c7aa0 0x7feec8153790 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7feec000c7f0 tx=0x7feec000ccb0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feece963640 1 -- 172.21.15.136:0/807864246 reap_dead start 2024-04-25T11:45:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 shutdown_connections 2024-04-25T11:45:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 >> 172.21.15.136:0/807864246 conn(0x7feec806d950 msgr2=0x7feec8142130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 shutdown_connections 2024-04-25T11:45:08.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.079+0000 7feed03c6640 1 -- 172.21.15.136:0/807864246 wait complete. 2024-04-25T11:45:08.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T11:45:08.222 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 Processor -- start 2024-04-25T11:45:08.223 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 -- start start 2024-04-25T11:45:08.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc16806cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 0x7fc16815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1680c8a20 con 0x7fc1680c9b50 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc16806c670 con 0x7fc16806d040 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16cad9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc1680c8d00 con 0x7fc1680ca010 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc166d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 0x7fc16815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc16806cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc166d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 0x7fc16815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:39342/0 (socket says 172.21.15.136:39342) 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc166d9d640 1 -- 172.21.15.136:0/2803715306 learned_addr learned my addr 172.21.15.136:0/2803715306 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc165d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc1680c9b50 0x7fc16815bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:41186/0 (socket says 172.21.15.136:41186) 2024-04-25T11:45:08.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16659c640 1 -- 172.21.15.136:0/2803715306 >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 msgr2=0x7fc16815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16659c640 1 --2- 172.21.15.136:0/2803715306 >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 0x7fc16815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16659c640 1 -- 172.21.15.136:0/2803715306 >> v1:172.21.15.157:6789/0 conn(0x7fc1680c9b50 legacy=0x7fc16815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.239+0000 7fc16659c640 1 -- 172.21.15.136:0/2803715306 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc168074d10 con 0x7fc1680ca010 2024-04-25T11:45:08.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16659c640 1 --2- 172.21.15.136:0/2803715306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc16806cb00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097c0 tx=0x7fc15801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f70c072991281e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16559a640 1 -- 172.21.15.136:0/2803715306 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1580b0070 con 0x7fc1680ca010 2024-04-25T11:45:08.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16559a640 1 -- 172.21.15.136:0/2803715306 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1580ab600 con 0x7fc1680ca010 2024-04-25T11:45:08.228 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16559a640 1 -- 172.21.15.136:0/2803715306 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1580ab9d0 con 0x7fc1680ca010 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 -- 172.21.15.136:0/2803715306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 msgr2=0x7fc16806cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 --2- 172.21.15.136:0/2803715306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc16806cb00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc1580097c0 tx=0x7fc15801c4a0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 -- 172.21.15.136:0/2803715306 shutdown_connections 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 --2- 172.21.15.136:0/2803715306 >> v2:172.21.15.157:3300/0 conn(0x7fc16806d040 0x7fc16815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 --2- 172.21.15.136:0/2803715306 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc16806cb00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 -- 172.21.15.136:0/2803715306 >> 172.21.15.136:0/2803715306 conn(0x7fc1680ca5a0 msgr2=0x7fc1680ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 -- 172.21.15.136:0/2803715306 shutdown_connections 2024-04-25T11:45:08.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.243+0000 7fc16cad9640 1 -- 172.21.15.136:0/2803715306 wait complete. 2024-04-25T11:45:08.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 Processor -- start 2024-04-25T11:45:08.231 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- start start 2024-04-25T11:45:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc16806d040 0x7fc168155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc168147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc16815cdb0 con 0x7fc1680c9b50 2024-04-25T11:45:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc16815cc30 con 0x7fc16806d040 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc168169140 con 0x7fc1680ca010 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44744/0 (socket says 172.21.15.136:44744) 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 -- 172.21.15.136:0/1287726213 learned_addr learned my addr 172.21.15.136:0/1287726213 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc166d9d640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc168147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16659c640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc16806d040 0x7fc168155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 -- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 msgr2=0x7fc168147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc168147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 -- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc16806d040 msgr2=0x7fc168155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc16806d040 0x7fc168155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 -- 172.21.15.136:0/1287726213 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc168155850 con 0x7fc1680c9b50 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc165d9b640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c340 tx=0x7fc15c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.234 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0074e0 con 0x7fc1680c9b50 2024-04-25T11:45:08.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15c0076a0 con 0x7fc1680c9b50 2024-04-25T11:45:08.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1681f5b70 con 0x7fc1680c9b50 2024-04-25T11:45:08.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004a30 con 0x7fc1680c9b50 2024-04-25T11:45:08.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1681f6130 con 0x7fc1680c9b50 2024-04-25T11:45:08.235 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.247+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1681f58e0 con 0x7fc1680c9b50 2024-04-25T11:45:08.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.251+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc15c01a020 con 0x7fc1680c9b50 2024-04-25T11:45:08.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.251+0000 7fc156ffd640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 0x7fc148071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.251+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fc15c08f250 con 0x7fc1680c9b50 2024-04-25T11:45:08.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.251+0000 7fc16659c640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 0x7fc148071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.236 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.251+0000 7fc16659c640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 0x7fc148071910 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc1580ad000 tx=0x7fc158008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.240 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.255+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc15c05b880 con 0x7fc1680c9b50 2024-04-25T11:45:08.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.415+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fc16806a350 con 0x7fc14806f400 2024-04-25T11:45:08.402 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc156ffd640 1 -- 172.21.15.136:0/1287726213 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc16806a350 con 0x7fc14806f400 2024-04-25T11:45:08.402 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 msgr2=0x7fc148071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 0x7fc148071910 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc1580ad000 tx=0x7fc158008040 comp rx=0 tx=0).stop 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 msgr2=0x7fc1680ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00c340 tx=0x7fc15c00c800 comp rx=0 tx=0).stop 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 shutdown_connections 2024-04-25T11:45:08.405 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fc14806f400 0x7fc148071910 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc1680ca010 0x7fc168147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1680c9b50 0x7fc1680ccb30 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 --2- 172.21.15.136:0/1287726213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc16806d040 0x7fc168155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 >> 172.21.15.136:0/1287726213 conn(0x7fc1680ca5a0 msgr2=0x7fc168075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 shutdown_connections 2024-04-25T11:45:08.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.419+0000 7fc16cad9640 1 -- 172.21.15.136:0/1287726213 wait complete. 2024-04-25T11:45:08.424 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T11:45:08.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:08.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 Processor -- start 2024-04-25T11:45:08.568 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 -- start start 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1f80cb800 0x7ff1f80ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f815a130 0x7ff1f815a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f80c5220 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f80c66a0 con 0x7ff1f80cb800 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1fd269640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f80c5500 con 0x7ff1f815a130 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1f80cb800 0x7ff1f80ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1f80cb800 0x7ff1f80ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44746/0 (socket says 172.21.15.136:44746) 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/4193670913 learned_addr learned my addr 172.21.15.136:0/4193670913 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/4193670913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f815a130 msgr2=0x7ff1f815a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 --2- 172.21.15.136:0/4193670913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f815a130 0x7ff1f815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/4193670913 >> v1:172.21.15.157:6789/0 conn(0x7ff1f80ca7d0 legacy=0x7ff1f80c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.583+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/4193670913 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f80c57e0 con 0x7ff1f80cb800 2024-04-25T11:45:08.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1f6ffd640 1 --2- 172.21.15.136:0/4193670913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80cb800 0x7ff1f80ca290 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8071690 tx=0x7ff1e801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aa247749d970f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1f67fc640 1 -- 172.21.15.136:0/4193670913 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80b5070 con 0x7ff1f80cb800 2024-04-25T11:45:08.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1f67fc640 1 -- 172.21.15.136:0/4193670913 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e80a89b0 con 0x7ff1f80cb800 2024-04-25T11:45:08.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1f67fc640 1 -- 172.21.15.136:0/4193670913 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80a8d80 con 0x7ff1f80cb800 2024-04-25T11:45:08.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 -- 172.21.15.136:0/4193670913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80cb800 msgr2=0x7ff1f80ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 --2- 172.21.15.136:0/4193670913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80cb800 0x7ff1f80ca290 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8071690 tx=0x7ff1e801ecf0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 -- 172.21.15.136:0/4193670913 shutdown_connections 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 --2- 172.21.15.136:0/4193670913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f815a130 0x7ff1f815a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 --2- 172.21.15.136:0/4193670913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80cb800 0x7ff1f80ca290 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 -- 172.21.15.136:0/4193670913 >> 172.21.15.136:0/4193670913 conn(0x7ff1f80726d0 msgr2=0x7ff1f8072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 -- 172.21.15.136:0/4193670913 shutdown_connections 2024-04-25T11:45:08.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.587+0000 7ff1fd269640 1 -- 172.21.15.136:0/4193670913 wait complete. 2024-04-25T11:45:08.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 Processor -- start 2024-04-25T11:45:08.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- start start 2024-04-25T11:45:08.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 0x7ff1f8157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80cb800 0x7ff1f80cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815a130 0x7ff1f81e4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f815aa80 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f80c5e30 con 0x7ff1f815a130 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f80c4bc0 con 0x7ff1f80cb800 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 0x7ff1f8157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 0x7ff1f8157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44752/0 (socket says 172.21.15.136:44752) 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/2379555567 learned_addr learned my addr 172.21.15.136:0/2379555567 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f77fe640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815a130 0x7ff1f81e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80cb800 msgr2=0x7ff1f80cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1effff640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80cb800 0x7ff1f80cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1f80cb800 0x7ff1f80cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815a130 msgr2=0x7ff1f81e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1f815a130 0x7ff1f81e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f81e43c0 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1f6ffd640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 0x7ff1f8157a00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff1e801f590 tx=0x7ff1e801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80b5040 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e80be070 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f80b42a0 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80b0af0 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f80b4820 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.591+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1f80b4ee0 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1e80b0c90 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1d806f400 0x7ff1d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7ff1e812db00 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7ff1d8079880 0x7ff1d807bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1e80a9e50 con 0x7ff1d8079880 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff1e812df90 con 0x7ff1f80ca7d0 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1effff640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1d806f400 0x7ff1d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1f77fe640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7ff1d8079880 0x7ff1d807bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1effff640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1d806f400 0x7ff1d8071910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff1f81f49b0 tx=0x7ff1e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1f77fe640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7ff1d8079880 0x7ff1d807bd70 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.595+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== osd.5 v2:172.21.15.157:6800/4215140907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1e80a9e50 con 0x7ff1d8079880 2024-04-25T11:45:08.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 --> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff1f8157f40 con 0x7ff1d8079880 2024-04-25T11:45:08.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1ef7fe640 1 -- 172.21.15.136:0/2379555567 <== osd.5 v2:172.21.15.157:6800/4215140907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff1f8157f40 con 0x7ff1d8079880 2024-04-25T11:45:08.596 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:08.596 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:08.596 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:08.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7ff1d8079880 msgr2=0x7ff1d807bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6800/4215140907,v1:172.21.15.157:6801/4215140907] conn(0x7ff1d8079880 0x7ff1d807bd70 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1d806f400 msgr2=0x7ff1d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff1d806f400 0x7ff1d8071910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff1f81f49b0 tx=0x7ff1e0008040 comp rx=0 tx=0).stop 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 msgr2=0x7ff1f8157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 --2- 172.21.15.136:0/2379555567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1f80ca7d0 0x7ff1f8157a00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7ff1e801f590 tx=0x7ff1e801ba90 comp rx=0 tx=0).stop 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1f77fe640 1 -- 172.21.15.136:0/2379555567 reap_dead start 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 shutdown_connections 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 >> 172.21.15.136:0/2379555567 conn(0x7ff1f80726d0 msgr2=0x7ff1f8072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 shutdown_connections 2024-04-25T11:45:08.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.611+0000 7ff1fd269640 1 -- 172.21.15.136:0/2379555567 wait complete. 2024-04-25T11:45:08.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T11:45:08.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.771+0000 7f469c7b2640 1 Processor -- start 2024-04-25T11:45:08.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.771+0000 7f469c7b2640 1 -- start start 2024-04-25T11:45:08.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469c7b2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f469415aae0 0x7f46940c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469c7b2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940c9af0 con 0x7f46940c79e0 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469c7b2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940cb220 con 0x7f46940c96e0 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469c7b2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f469406cc10 con 0x7f469415aae0 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f469415aae0 0x7f46940c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f469415aae0 0x7f46940c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44758/0 (socket says 172.21.15.136:44758) 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 -- 172.21.15.136:0/3470624092 learned_addr learned my addr 172.21.15.136:0/3470624092 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f4699d4d640 1 --1- 172.21.15.136:0/3470624092 >> v1:172.21.15.136:6789/0 conn(0x7f46940c79e0 0x7f469415a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46520/0 (socket says 172.21.15.136:46520) 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 -- 172.21.15.136:0/3470624092 >> v1:172.21.15.157:6789/0 conn(0x7f46940c96e0 legacy=0x7f46941540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469954c640 1 -- 172.21.15.136:0/3470624092 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385954910 0 0) 0x7f46940c9af0 con 0x7f46940c79e0 2024-04-25T11:45:08.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 -- 172.21.15.136:0/3470624092 >> v1:172.21.15.136:6789/0 conn(0x7f46940c79e0 legacy=0x7f469415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.775+0000 7f469ad4f640 1 -- 172.21.15.136:0/3470624092 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46940cb500 con 0x7f469415aae0 2024-04-25T11:45:08.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469ad4f640 1 --2- 172.21.15.136:0/3470624092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f469415aae0 0x7f46940c7df0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f469406cd90 tx=0x7f468c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5e4da332fe3220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469954c640 1 -- 172.21.15.136:0/3470624092 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c0a56e0 con 0x7f469415aae0 2024-04-25T11:45:08.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469954c640 1 -- 172.21.15.136:0/3470624092 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f468c0a5880 con 0x7f469415aae0 2024-04-25T11:45:08.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469954c640 1 -- 172.21.15.136:0/3470624092 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468c0a5c50 con 0x7f469415aae0 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- 172.21.15.136:0/3470624092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f469415aae0 msgr2=0x7f46940c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 --2- 172.21.15.136:0/3470624092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f469415aae0 0x7f46940c7df0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f469406cd90 tx=0x7f468c01ca10 comp rx=0 tx=0).stop 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- 172.21.15.136:0/3470624092 shutdown_connections 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 --2- 172.21.15.136:0/3470624092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f469415aae0 0x7f46940c7df0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- 172.21.15.136:0/3470624092 >> 172.21.15.136:0/3470624092 conn(0x7f469406f9d0 msgr2=0x7f469406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- 172.21.15.136:0/3470624092 shutdown_connections 2024-04-25T11:45:08.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- 172.21.15.136:0/3470624092 wait complete. 2024-04-25T11:45:08.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 Processor -- start 2024-04-25T11:45:08.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.779+0000 7f469c7b2640 1 -- start start 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46940c96e0 0x7f4694158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f469415aae0 0x7f46940c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f469406d5c0 con 0x7f46940c79e0 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f469406d440 con 0x7f46940c96e0 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4694168540 con 0x7f469415aae0 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44772/0 (socket says 172.21.15.136:44772) 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 -- 172.21.15.136:0/2643731382 learned_addr learned my addr 172.21.15.136:0/2643731382 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 -- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f469415aae0 msgr2=0x7f46940c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469ad4f640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f469415aae0 0x7f46940c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f4699d4d640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46940c96e0 0x7f4694158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f469415aae0 0x7f46940c6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 -- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46940c96e0 msgr2=0x7f4694158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46940c96e0 0x7f4694158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 -- 172.21.15.136:0/2643731382 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469413e820 con 0x7f46940c79e0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469a54e640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f469000c3f0 tx=0x7f469000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690017070 con 0x7f46940c79e0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46900052d0 con 0x7f46940c79e0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46940cd920 con 0x7f46940c79e0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690003840 con 0x7f46940c79e0 2024-04-25T11:45:08.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46940cdee0 con 0x7f46940c79e0 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469406a350 con 0x7f46940c79e0 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f469001a070 con 0x7f46940c79e0 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 0x7f4678071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f469008eef0 con 0x7f46940c79e0 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.783+0000 7f4699d4d640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 0x7f4678071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:08.771 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.787+0000 7f4699d4d640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 0x7f4678071910 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4684007fd0 tx=0x7f4684009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:08.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.787+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4690005480 con 0x7f46940c79e0 2024-04-25T11:45:08.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f46940cd3c0 con 0x7f467806f400 2024-04-25T11:45:08.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f468affd640 1 -- 172.21.15.136:0/2643731382 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f46840070d0 con 0x7f467806f400 2024-04-25T11:45:08.944 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T11:45:08.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 msgr2=0x7f4678071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 0x7f4678071910 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4684007fd0 tx=0x7f4684009040 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 msgr2=0x7f469413e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f469000c3f0 tx=0x7f469000c8b0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 shutdown_connections 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f467806f400 0x7f4678071910 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f469415aae0 0x7f46940c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46940c96e0 0x7f4694158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 --2- 172.21.15.136:0/2643731382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46940c79e0 0x7f469413e2e0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.959+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 >> 172.21.15.136:0/2643731382 conn(0x7f469406f9d0 msgr2=0x7f469414edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.963+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 shutdown_connections 2024-04-25T11:45:08.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:08.963+0000 7f469c7b2640 1 -- 172.21.15.136:0/2643731382 wait complete. 2024-04-25T11:45:08.965 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T11:45:08.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:09.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.119+0000 7f57003ad640 1 Processor -- start 2024-04-25T11:45:09.106 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.119+0000 7f57003ad640 1 -- start start 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f57003ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56f8154260 0x7f56f806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f57003ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f8164850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f57003ad640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f806c710 con 0x7f56f8159ed0 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f57003ad640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f56f8154670 con 0x7f56f8154260 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f57003ad640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f56f806ccd0 con 0x7f56f815a2e0 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56f8154260 0x7f56f806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56f8154260 0x7f56f806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44776/0 (socket says 172.21.15.136:44776) 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fd948640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f56f8159ed0 0x7f56f80ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46532/0 (socket says 172.21.15.136:46532) 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 -- 172.21.15.136:0/3625496941 learned_addr learned my addr 172.21.15.136:0/3625496941 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe94a640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f8164850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.109 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 -- 172.21.15.136:0/3625496941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 msgr2=0x7f56f8164850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f8164850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 -- 172.21.15.136:0/3625496941 >> v1:172.21.15.136:6789/0 conn(0x7f56f8159ed0 legacy=0x7f56f80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe149640 1 -- 172.21.15.136:0/3625496941 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f806c9f0 con 0x7f56f8154260 2024-04-25T11:45:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fe94a640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f8164850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.123+0000 7f56fd147640 1 -- 172.21.15.136:0/3625496941 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513773785 0 0) 0x7f56f806c710 con 0x7f56f8159ed0 2024-04-25T11:45:09.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f56fe149640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8154260 0x7f56f806d0b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009870 tx=0x7f56f401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc746d3b4d907852 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.111 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f56fd147640 1 -- 172.21.15.136:0/3625496941 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f40b0070 con 0x7f56f8154260 2024-04-25T11:45:09.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f56fd147640 1 -- 172.21.15.136:0/3625496941 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f40ab600 con 0x7f56f8154260 2024-04-25T11:45:09.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f56fd147640 1 -- 172.21.15.136:0/3625496941 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f40ab9d0 con 0x7f56f8154260 2024-04-25T11:45:09.112 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- 172.21.15.136:0/3625496941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8154260 msgr2=0x7f56f806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8154260 0x7f56f806d0b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009870 tx=0x7f56f401c4d0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- 172.21.15.136:0/3625496941 shutdown_connections 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f8164850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 --2- 172.21.15.136:0/3625496941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8154260 0x7f56f806d0b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- 172.21.15.136:0/3625496941 >> 172.21.15.136:0/3625496941 conn(0x7f56f806d950 msgr2=0x7f56f806dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- 172.21.15.136:0/3625496941 shutdown_connections 2024-04-25T11:45:09.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- 172.21.15.136:0/3625496941 wait complete. 2024-04-25T11:45:09.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 Processor -- start 2024-04-25T11:45:09.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.127+0000 7f57003ad640 1 -- start start 2024-04-25T11:45:09.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56f8154260 0x7f56f8155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 0x7f56f815b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f814f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f56f8074dd0 con 0x7f56f8159ed0 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f56f815a6f0 con 0x7f56f8154260 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f56f80c8770 con 0x7f56f815a2e0 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 0x7f56f815b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 0x7f56f815b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44788/0 (socket says 172.21.15.136:44788) 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 -- 172.21.15.136:0/1010324928 learned_addr learned my addr 172.21.15.136:0/1010324928 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fe94a640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f814f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fe149640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56f8154260 0x7f56f8155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 -- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 msgr2=0x7f56f814f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56f815a2e0 0x7f56f814f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 -- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56f8154260 msgr2=0x7f56f8155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56f8154260 0x7f56f8155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.118 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f8155fa0 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fe149640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56f8154260 0x7f56f8155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56fd948640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 0x7f56f815b260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c480 tx=0x7f56e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8018070 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e80074e0 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e80049f0 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f8155580 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f81f6ef0 con 0x7f56f8159ed0 2024-04-25T11:45:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f56f806a350 con 0x7f56f8159ed0 2024-04-25T11:45:09.120 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.131+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f56e801b070 con 0x7f56f8159ed0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f56dc06f430 0x7f56dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f56e808f010 con 0x7f56f8159ed0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7f56dc0798b0 0x7f56dc07bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56dc07c470 con 0x7f56dc0798b0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f56e808f4a0 con 0x7f56f8159ed0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56fe149640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f56dc06f430 0x7f56dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56fe94a640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7f56dc0798b0 0x7f56dc07bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56fe149640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f56dc06f430 0x7f56dc071940 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56f40ad000 tx=0x7f56f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.121 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56fe94a640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7f56dc0798b0 0x7f56dc07bda0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.122 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.135+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== osd.6 v2:172.21.15.157:6824/224466234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f56dc07c470 con 0x7f56dc0798b0 2024-04-25T11:45:09.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 --> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f56f81f69c0 con 0x7f56dc0798b0 2024-04-25T11:45:09.135 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f56eeffd640 1 -- 172.21.15.136:0/1010324928 <== osd.6 v2:172.21.15.157:6824/224466234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f56f81f69c0 con 0x7f56dc0798b0 2024-04-25T11:45:09.136 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:09.136 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:09.136 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7f56dc0798b0 msgr2=0x7f56dc07bda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6824/224466234,v1:172.21.15.157:6825/224466234] conn(0x7f56dc0798b0 0x7f56dc07bda0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f56dc06f430 msgr2=0x7f56dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f56dc06f430 0x7f56dc071940 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56f40ad000 tx=0x7f56f4008040 comp rx=0 tx=0).stop 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 msgr2=0x7f56f815b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 --2- 172.21.15.136:0/1010324928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56f8159ed0 0x7f56f815b260 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c480 tx=0x7f56e800c940 comp rx=0 tx=0).stop 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f56fe94a640 1 -- 172.21.15.136:0/1010324928 reap_dead start 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 shutdown_connections 2024-04-25T11:45:09.137 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 >> 172.21.15.136:0/1010324928 conn(0x7f56f806d950 msgr2=0x7f56f815a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 shutdown_connections 2024-04-25T11:45:09.138 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.151+0000 7f57003ad640 1 -- 172.21.15.136:0/1010324928 wait complete. 2024-04-25T11:45:09.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 Processor -- start 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 -- start start 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe7dc16cc40 0x7fe7dc165e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc073970 0x7fe7dc06fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7dc16d050 con 0x7fe7dc166390 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7fe7dc1667a0 con 0x7fe7dc16cc40 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e25e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7dc08d2b0 con 0x7fe7dc073970 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7dbfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe7dc16cc40 0x7fe7dc165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7db7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe7dc166390 0x7fe7dc073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46548/0 (socket says 172.21.15.136:46548) 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7db7fe640 1 -- 172.21.15.136:0/1516015643 learned_addr learned my addr 172.21.15.136:0/1516015643 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7dbfff640 1 --2- 172.21.15.136:0/1516015643 >> v2:172.21.15.136:3300/0 conn(0x7fe7dc16cc40 0x7fe7dc165e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44794/0 (socket says 172.21.15.136:44794) 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771038080 0 0) 0x7fe7dc16d050 con 0x7fe7dc166390 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.303+0000 7fe7e0b7f640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc073970 0x7fe7dc06fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7c4003510 con 0x7fe7dc166390 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7dbfff640 1 -- 172.21.15.136:0/1516015643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc073970 msgr2=0x7fe7dc06fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7dbfff640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc073970 0x7fe7dc06fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4010848489 0 0) 0x7fe7c4003510 con 0x7fe7dc166390 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7dbfff640 1 -- 172.21.15.136:0/1516015643 >> v1:172.21.15.136:6789/0 conn(0x7fe7dc166390 legacy=0x7fe7dc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7dbfff640 1 -- 172.21.15.136:0/1516015643 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc08d560 con 0x7fe7dc16cc40 2024-04-25T11:45:09.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7dbfff640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc16cc40 0x7fe7dc165e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc008840 tx=0x7fe7cc01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb524789207ddb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0001680 con 0x7fe7dc16cc40 2024-04-25T11:45:09.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00ae50 con 0x7fe7dc16cc40 2024-04-25T11:45:09.310 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.323+0000 7fe7daffd640 1 -- 172.21.15.136:0/1516015643 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0a8a40 con 0x7fe7dc16cc40 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1516015643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc16cc40 msgr2=0x7fe7dc165e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc16cc40 0x7fe7dc165e50 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc008840 tx=0x7fe7cc01ecd0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1516015643 shutdown_connections 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc073970 0x7fe7dc06fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1516015643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc16cc40 0x7fe7dc165e50 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1516015643 >> 172.21.15.136:0/1516015643 conn(0x7fe7dc0c9d50 msgr2=0x7fe7dc0ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1516015643 shutdown_connections 2024-04-25T11:45:09.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1516015643 wait complete. 2024-04-25T11:45:09.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 Processor -- start 2024-04-25T11:45:09.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- start start 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7dc073970 0x7fe7dc1ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc16cc40 0x7fe7dc162980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc072a60 con 0x7fe7dc166390 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc08db20 con 0x7fe7dc073970 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7e25e2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7dc072d40 con 0x7fe7dc16cc40 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7db7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7db7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44798/0 (socket says 172.21.15.136:44798) 2024-04-25T11:45:09.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.327+0000 7fe7db7fe640 1 -- 172.21.15.136:0/1869416358 learned_addr learned my addr 172.21.15.136:0/1869416358 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 -- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc16cc40 msgr2=0x7fe7dc162980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc16cc40 0x7fe7dc162980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 -- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7dc073970 msgr2=0x7fe7dc1ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7dc073970 0x7fe7dc1ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 -- 172.21.15.136:0/1869416358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc16a920 con 0x7fe7dc166390 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7db7fe640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00017f0 tx=0x7fe7d000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0005ae0 con 0x7fe7dc166390 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0005c80 con 0x7fe7dc166390 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0007290 con 0x7fe7dc166390 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7dc16b6d0 con 0x7fe7dc166390 2024-04-25T11:45:09.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7dc16bc70 con 0x7fe7dc166390 2024-04-25T11:45:09.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7dc06a350 con 0x7fe7dc166390 2024-04-25T11:45:09.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe7d0007430 con 0x7fe7dc166390 2024-04-25T11:45:09.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 0x7fe7c8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fe7d0090330 con 0x7fe7dc166390 2024-04-25T11:45:09.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7dbfff640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 0x7fe7c8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.331+0000 7fe7dbfff640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 0x7fe7c8071910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0b1000 tx=0x7fe7cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.321 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.335+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe7d0007720 con 0x7fe7dc166390 2024-04-25T11:45:09.474 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.487+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fe7dc16b470 con 0x7fe7c806f400 2024-04-25T11:45:09.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7d8ff9640 1 -- 172.21.15.136:0/1869416358 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe7cc0a98b0 con 0x7fe7c806f400 2024-04-25T11:45:09.475 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T11:45:09.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 msgr2=0x7fe7c8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 0x7fe7c8071910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0b1000 tx=0x7fe7cc008040 comp rx=0 tx=0).stop 2024-04-25T11:45:09.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 msgr2=0x7fe7dc16a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00017f0 tx=0x7fe7d000cbe0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 shutdown_connections 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fe7c806f400 0x7fe7c8071910 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7dc16cc40 0x7fe7dc162980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7dc166390 0x7fe7dc16a3e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 --2- 172.21.15.136:0/1869416358 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7dc073970 0x7fe7dc1ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 >> 172.21.15.136:0/1869416358 conn(0x7fe7dc0c9d50 msgr2=0x7fe7dc164150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 shutdown_connections 2024-04-25T11:45:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.491+0000 7fe7e25e2640 1 -- 172.21.15.136:0/1869416358 wait complete. 2024-04-25T11:45:09.496 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T11:45:09.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T11:45:09.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.663+0000 7f072386f640 1 Processor -- start 2024-04-25T11:45:09.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.663+0000 7f072386f640 1 -- start start 2024-04-25T11:45:09.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072386f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f071c0726d0 0x7f071c072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072386f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072386f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071c0c5390 con 0x7f071c154160 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072386f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f071c0cccc0 con 0x7f071c0726d0 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072386f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f071c15a0e0 con 0x7f071c15d8a0 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f071c0726d0 0x7f071c072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072160b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f071c154160 0x7f071c0c9f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:41214/0 (socket says 172.21.15.136:41214) 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f071c0726d0 0x7f071c072ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.136:39394/0 (socket says 172.21.15.136:39394) 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f072160b640 1 -- 172.21.15.136:0/3563922689 learned_addr learned my addr 172.21.15.136:0/3563922689 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0721e0c640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 -- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 msgr2=0x7f071c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:09.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 -- 172.21.15.136:0/3563922689 >> v1:172.21.15.157:6789/0 conn(0x7f071c154160 legacy=0x7f071c0c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 -- 172.21.15.136:0/3563922689 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c15a3c0 con 0x7f071c0726d0 2024-04-25T11:45:09.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.667+0000 7f0720e0a640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c072ae0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f07180097c0 tx=0x7f071801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba166e7845066467 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f0713fff640 1 -- 172.21.15.136:0/3563922689 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180b5070 con 0x7f071c0726d0 2024-04-25T11:45:09.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f0713fff640 1 -- 172.21.15.136:0/3563922689 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07180a8990 con 0x7f071c0726d0 2024-04-25T11:45:09.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f0713fff640 1 -- 172.21.15.136:0/3563922689 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180a8d60 con 0x7f071c0726d0 2024-04-25T11:45:09.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 msgr2=0x7f071c072ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c072ae0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f07180097c0 tx=0x7f071801ec90 comp rx=0 tx=0).stop 2024-04-25T11:45:09.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- 172.21.15.136:0/3563922689 shutdown_connections 2024-04-25T11:45:09.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 --2- 172.21.15.136:0/3563922689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c072ae0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- 172.21.15.136:0/3563922689 >> 172.21.15.136:0/3563922689 conn(0x7f071c072000 msgr2=0x7f071c070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- 172.21.15.136:0/3563922689 shutdown_connections 2024-04-25T11:45:09.656 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- 172.21.15.136:0/3563922689 wait complete. 2024-04-25T11:45:09.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 Processor -- start 2024-04-25T11:45:09.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.671+0000 7f072386f640 1 -- start start 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c1f0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 0x7f071c0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f071c15ac60 con 0x7f071c154160 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f071c15a980 con 0x7f071c0726d0 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f071c154570 con 0x7f071c15d8a0 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 0x7f071c0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 0x7f071c0b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44812/0 (socket says 172.21.15.136:44812) 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 -- 172.21.15.136:0/3077812732 learned_addr learned my addr 172.21.15.136:0/3077812732 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072160b640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c1f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0721e0c640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 -- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 msgr2=0x7f071c15ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f071c15d8a0 0x7f071c15ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 -- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 msgr2=0x7f071c1f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c1f0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c1f3860 con 0x7f071c154160 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072160b640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f071c0726d0 0x7f071c1f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0720e0a640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 0x7f071c0b3c10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f07180b3040 tx=0x7f071801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180bf070 con 0x7f071c154160 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07180b0680 con 0x7f071c154160 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07180b5040 con 0x7f071c154160 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c1f3b40 con 0x7f071c154160 2024-04-25T11:45:09.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c0b4150 con 0x7f071c154160 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.675+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f071c06a3c0 con 0x7f071c154160 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f07180b0a70 con 0x7f071c154160 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f070006f400 0x7f0700071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f071812cc20 con 0x7f071c154160 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7f0700079880 0x7f070007bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07180044a0 con 0x7f0700079880 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f071812d0b0 con 0x7f071c154160 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f072160b640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f070006f400 0x7f0700071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0721e0c640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7f0700079880 0x7f070007bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f072160b640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f070006f400 0x7f0700071910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f070c005890 tx=0x7f070c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0721e0c640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7f0700079880 0x7f070007bd70 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.679+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== osd.7 v2:172.21.15.157:6809/498117293 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07180044a0 con 0x7f0700079880 2024-04-25T11:45:09.679 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 --> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f071c0b48d0 con 0x7f0700079880 2024-04-25T11:45:09.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f0711ffb640 1 -- 172.21.15.136:0/3077812732 <== osd.7 v2:172.21.15.157:6809/498117293 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f071c0b48d0 con 0x7f0700079880 2024-04-25T11:45:09.680 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-25T11:45:09.680 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T11:45:09.680 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-25T11:45:09.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7f0700079880 msgr2=0x7f070007bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6809/498117293,v1:172.21.15.157:6811/498117293] conn(0x7f0700079880 0x7f070007bd70 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.681 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f070006f400 msgr2=0x7f0700071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f070006f400 0x7f0700071910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f070c005890 tx=0x7f070c009040 comp rx=0 tx=0).stop 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 msgr2=0x7f071c0b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 --2- 172.21.15.136:0/3077812732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f071c154160 0x7f071c0b3c10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f07180b3040 tx=0x7f071801ba30 comp rx=0 tx=0).stop 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f0721e0c640 1 -- 172.21.15.136:0/3077812732 reap_dead start 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 shutdown_connections 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 >> 172.21.15.136:0/3077812732 conn(0x7f071c072000 msgr2=0x7f071c152a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 shutdown_connections 2024-04-25T11:45:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.695+0000 7f072386f640 1 -- 172.21.15.136:0/3077812732 wait complete. 2024-04-25T11:45:09.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T11:45:09.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 Processor -- start 2024-04-25T11:45:09.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 -- start start 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d814f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08d80ce260 0x7f08d80ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08d80cbad0 con 0x7f08d80ca010 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f08d80c8550 con 0x7f08d80ce260 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08dd413640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08d80c7c20 con 0x7f08d80c8cd0 2024-04-25T11:45:09.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08cffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08d80ce260 0x7f08d80ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08d77fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f08d80ca010 0x7f08d80ca420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46560/0 (socket says 172.21.15.136:46560) 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08d77fe640 1 -- 172.21.15.136:0/824837845 learned_addr learned my addr 172.21.15.136:0/824837845 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08cffff640 1 -- 172.21.15.136:0/824837845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 msgr2=0x7f08d814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08d6ffd640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d814f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08d67fc640 1 -- 172.21.15.136:0/824837845 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2539511253 0 0) 0x7f08d80cbad0 con 0x7f08d80ca010 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08cffff640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d814f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08cffff640 1 -- 172.21.15.136:0/824837845 >> v1:172.21.15.136:6789/0 conn(0x7f08d80ca010 legacy=0x7f08d80ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.843 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.855+0000 7f08cffff640 1 -- 172.21.15.136:0/824837845 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d80c7f00 con 0x7f08d80ce260 2024-04-25T11:45:09.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08cffff640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ce260 0x7f08d80ce670 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f08d80c86d0 tx=0x7f08c001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ce7ad64a8c020c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08d67fc640 1 -- 172.21.15.136:0/824837845 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00b5070 con 0x7f08d80ce260 2024-04-25T11:45:09.844 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08d67fc640 1 -- 172.21.15.136:0/824837845 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c00a8990 con 0x7f08d80ce260 2024-04-25T11:45:09.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08d67fc640 1 -- 172.21.15.136:0/824837845 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00a8d60 con 0x7f08d80ce260 2024-04-25T11:45:09.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 -- 172.21.15.136:0/824837845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ce260 msgr2=0x7f08d80ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ce260 0x7f08d80ce670 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f08d80c86d0 tx=0x7f08c001ec90 comp rx=0 tx=0).stop 2024-04-25T11:45:09.845 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 -- 172.21.15.136:0/824837845 shutdown_connections 2024-04-25T11:45:09.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ce260 0x7f08d80ce670 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 --2- 172.21.15.136:0/824837845 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d814f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 -- 172.21.15.136:0/824837845 >> 172.21.15.136:0/824837845 conn(0x7f08d806f9d0 msgr2=0x7f08d806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:09.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 -- 172.21.15.136:0/824837845 shutdown_connections 2024-04-25T11:45:09.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.859+0000 7f08dd413640 1 -- 172.21.15.136:0/824837845 wait complete. 2024-04-25T11:45:09.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 Processor -- start 2024-04-25T11:45:09.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- start start 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d8155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 0x7f08d8153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f08d80cb720 con 0x7f08d80ca010 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08d814fbc0 con 0x7f08d80ce260 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08d8156310 con 0x7f08d80c8cd0 2024-04-25T11:45:09.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44828/0 (socket says 172.21.15.136:44828) 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 -- 172.21.15.136:0/206695186 learned_addr learned my addr 172.21.15.136:0/206695186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08d77fe640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 0x7f08d8153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08d6ffd640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d8155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 -- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 msgr2=0x7f08d8155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d8155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 -- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 msgr2=0x7f08d8153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 0x7f08d8153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 -- 172.21.15.136:0/206695186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d8155e40 con 0x7f08d80ca010 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08d77fe640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 0x7f08d8153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cffff640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f08c00b3040 tx=0x7f08c001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00b5040 con 0x7f08d80ca010 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c00bf070 con 0x7f08d80ca010 2024-04-25T11:45:09.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d806de80 con 0x7f08d80ca010 2024-04-25T11:45:09.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00b0a60 con 0x7f08d80ca010 2024-04-25T11:45:09.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d8154e20 con 0x7f08d80ca010 2024-04-25T11:45:09.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.863+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d806a350 con 0x7f08d80ca010 2024-04-25T11:45:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.867+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f08c00b0c00 con 0x7f08d80ca010 2024-04-25T11:45:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.867+0000 7f08cf7fe640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 0x7f08b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.867+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f08c012db80 con 0x7f08d80ca010 2024-04-25T11:45:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.867+0000 7f08d6ffd640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 0x7f08b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:09.853 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.867+0000 7f08d6ffd640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 0x7f08b8071910 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f08c800b7c0 tx=0x7f08c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:09.856 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:09.871+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08c00187f0 con 0x7f08d80ca010 2024-04-25T11:45:10.028 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.043+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f08d81645e0 con 0x7f08b806f400 2024-04-25T11:45:10.029 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.043+0000 7f08cf7fe640 1 -- 172.21.15.136:0/206695186 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f08d81645e0 con 0x7f08b806f400 2024-04-25T11:45:10.029 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T11:45:10.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 msgr2=0x7f08b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 0x7f08b8071910 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f08c800b7c0 tx=0x7f08c8008040 comp rx=0 tx=0).stop 2024-04-25T11:45:10.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 msgr2=0x7f08d814b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.031 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f08c00b3040 tx=0x7f08c001ba30 comp rx=0 tx=0).stop 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 shutdown_connections 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08d80ce260 0x7f08d8153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08d80ca010 0x7f08d814b7e0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f08b806f400 0x7f08b8071910 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 --2- 172.21.15.136:0/206695186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08d80c8cd0 0x7f08d8155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 >> 172.21.15.136:0/206695186 conn(0x7f08d806f9d0 msgr2=0x7f08d80ca420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 shutdown_connections 2024-04-25T11:45:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.047+0000 7f08dd413640 1 -- 172.21.15.136:0/206695186 wait complete. 2024-04-25T11:45:10.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:10.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 Processor -- start 2024-04-25T11:45:10.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 -- start start 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1a4c0c9b50 0x7f1a4c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 0x7f1a4c15c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4c0c8a20 con 0x7f1a4c0ca010 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1a4c06c670 con 0x7f1a4c0c9b50 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a537a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4c0c8d00 con 0x7f1a4c06d210 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a51545640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1a4c0ca010 0x7f1a4c06cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:46566/0 (socket says 172.21.15.136:46566) 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a51545640 1 -- 172.21.15.136:0/4113534132 learned_addr learned my addr 172.21.15.136:0/4113534132 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a51d46640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 0x7f1a4c15c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a50d44640 1 --2- 172.21.15.136:0/4113534132 >> v2:172.21.15.157:3300/0 conn(0x7f1a4c0c9b50 0x7f1a4c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194047026 0 0) 0x7f1a4c0c8a20 con 0x7f1a4c0ca010 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a38003530 con 0x7f1a4c0ca010 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2587269485 0 0) 0x7f1a38003530 con 0x7f1a4c0ca010 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 msgr2=0x7f1a4c15c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 0x7f1a4c15c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 >> v2:172.21.15.157:3300/0 conn(0x7f1a4c0c9b50 msgr2=0x7f1a4c15bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 --2- 172.21.15.136:0/4113534132 >> v2:172.21.15.157:3300/0 conn(0x7f1a4c0c9b50 0x7f1a4c15bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c074d10 con 0x7f1a4c0ca010 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f1a480016a0 con 0x7f1a4c0ca010 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 >> v1:172.21.15.136:6789/0 conn(0x7f1a4c0ca010 legacy=0x7f1a4c06cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a380039c0 0x7f1a38003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 0x7f1a38004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a38004d50 0x7f1a38005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c06c670 con 0x7f1a380039c0 2024-04-25T11:45:10.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c0c8d00 con 0x7f1a38004d50 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3800ba00 con 0x7f1a4c06d210 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.207+0000 7f1a51545640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a380039c0 0x7f1a38003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 -- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 msgr2=0x7f1a38004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c06d210 0x7f1a38004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 -- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a38004d50 msgr2=0x7f1a38005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a38004d50 0x7f1a38005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 -- 172.21.15.136:0/4113534132 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a38003530 con 0x7f1a380039c0 2024-04-25T11:45:10.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a51545640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a380039c0 0x7f1a38003e30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c068ac0 tx=0x7f1a4801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501a9271c0a003b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:10.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48007d70 con 0x7f1a380039c0 2024-04-25T11:45:10.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a480a9020 con 0x7f1a380039c0 2024-04-25T11:45:10.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.211+0000 7f1a43fff640 1 -- 172.21.15.136:0/4113534132 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48005ae0 con 0x7f1a380039c0 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- 172.21.15.136:0/4113534132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a380039c0 msgr2=0x7f1a38003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 --2- 172.21.15.136:0/4113534132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a380039c0 0x7f1a38003e30 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c068ac0 tx=0x7f1a4801cc20 comp rx=0 tx=0).stop 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a51d46640 1 -- 172.21.15.136:0/4113534132 reap_dead start 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- 172.21.15.136:0/4113534132 shutdown_connections 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- 172.21.15.136:0/4113534132 >> 172.21.15.136:0/4113534132 conn(0x7f1a4c0ca5a0 msgr2=0x7f1a4c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- 172.21.15.136:0/4113534132 shutdown_connections 2024-04-25T11:45:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- 172.21.15.136:0/4113534132 wait complete. 2024-04-25T11:45:10.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 Processor -- start 2024-04-25T11:45:10.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.215+0000 7f1a537a9640 1 -- start start 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4c14c170 0x7f1a4c1f9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c1f6290 0x7f1a4c149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c166cb0 con 0x7f1a4c14cba0 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c166b30 con 0x7f1a4c14c170 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4c166e30 con 0x7f1a4c1f6290 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44850/0 (socket says 172.21.15.136:44850) 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 -- 172.21.15.136:0/2320229792 learned_addr learned my addr 172.21.15.136:0/2320229792 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51545640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4c14c170 0x7f1a4c1f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a50d44640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c1f6290 0x7f1a4c149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 -- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c1f6290 msgr2=0x7f1a4c149030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c1f6290 0x7f1a4c149030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 -- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4c14c170 msgr2=0x7f1a4c1f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4c14c170 0x7f1a4c1f9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 -- 172.21.15.136:0/2320229792 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c0c4e90 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a51d46640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c06e920 tx=0x7f1a4400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400ba50 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a4401a070 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c0c5170 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400be30 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c1f5d70 con 0x7f1a4c14cba0 2024-04-25T11:45:10.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c06a350 con 0x7f1a4c14cba0 2024-04-25T11:45:10.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.219+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1a44013070 con 0x7f1a4c14cba0 2024-04-25T11:45:10.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.223+0000 7f1a41ffb640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 0x7f1a30071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:10.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.223+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f1a44090a20 con 0x7f1a4c14cba0 2024-04-25T11:45:10.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.223+0000 7f1a51545640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 0x7f1a30071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:10.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.223+0000 7f1a51545640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 0x7f1a30071910 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1a480aa000 tx=0x7f1a48001a70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:10.212 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.227+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1a4405d130 con 0x7f1a4c14cba0 2024-04-25T11:45:10.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.383+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a4c0cce40 con 0x7f1a3006f400 2024-04-25T11:45:10.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.407+0000 7f1a41ffb640 1 -- 172.21.15.136:0/2320229792 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+333255 (secure 0 0 0) 0x7f1a4c0cce40 con 0x7f1a3006f400 2024-04-25T11:45:10.397 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:10.399 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:10.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 msgr2=0x7f1a30071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.400 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 0x7f1a30071910 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1a480aa000 tx=0x7f1a48001a70 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 msgr2=0x7f1a4c14b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c06e920 tx=0x7f1a4400ee00 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 shutdown_connections 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a4c14cba0 0x7f1a4c14b840 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f1a3006f400 0x7f1a30071910 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4c1f6290 0x7f1a4c149030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 --2- 172.21.15.136:0/2320229792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4c14c170 0x7f1a4c1f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.411+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 >> 172.21.15.136:0/2320229792 conn(0x7f1a4c0ca5a0 msgr2=0x7f1a4c15bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.415+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 shutdown_connections 2024-04-25T11:45:10.401 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:10.415+0000 7f1a537a9640 1 -- 172.21.15.136:0/2320229792 wait complete. 2024-04-25T11:45:10.419 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2024-04-25T11:45:09.418848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15910156,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658432,"num_write_kb":15736463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8952152,"kb_used_data":34840,"kb_used_omap":12,"kb_used_meta":8917235,"kb_avail":746022568,"statfs":{"total":773094113280,"available":763927109632,"internally_reserved":0,"allocated":35676160,"data_stored":33593792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9131249202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4305580889,"num_objects":-1184,"num_object_clones":0,"num_object_copies":-4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004874"},"pg_stats":[{"pgid":"2.1f","version":"40'277","reported_seq":363,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.807296+0000","last_change":"2024-04-25T11:20:19.031966+0000","last_active":"2024-04-25T11:45:01.807296+0000","last_peered":"2024-04-25T11:45:01.807296+0000","last_clean":"2024-04-25T11:45:01.807296+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:45:01.807296+0000","last_undegraded":"2024-04-25T11:45:01.807296+0000","last_fullsized":"2024-04-25T11:45:01.807296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:28.188881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'7","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.683124+0000","last_change":"2024-04-25T11:20:20.023605+0000","last_active":"2024-04-25T11:45:01.683124+0000","last_peered":"2024-04-25T11:45:01.683124+0000","last_clean":"2024-04-25T11:45:01.683124+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:45:01.683124+0000","last_undegraded":"2024-04-25T11:45:01.683124+0000","last_fullsized":"2024-04-25T11:45:01.683124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5345","reported_seq":5451,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.956957+0000","last_change":"2024-04-25T11:20:24.165808+0000","last_active":"2024-04-25T11:45:01.956957+0000","last_peered":"2024-04-25T11:45:01.956957+0000","last_clean":"2024-04-25T11:45:01.956957+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:45:01.956957+0000","last_undegraded":"2024-04-25T11:45:01.956957+0000","last_fullsized":"2024-04-25T11:45:01.956957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22060,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'11","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.785904+0000","last_change":"2024-04-25T11:20:19.003929+0000","last_active":"2024-04-25T11:45:01.785904+0000","last_peered":"2024-04-25T11:45:01.785904+0000","last_clean":"2024-04-25T11:45:01.785904+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:45:01.785904+0000","last_undegraded":"2024-04-25T11:45:01.785904+0000","last_fullsized":"2024-04-25T11:45:01.785904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.908204+0000","last_change":"2024-04-25T11:20:20.035793+0000","last_active":"2024-04-25T11:45:01.908204+0000","last_peered":"2024-04-25T11:45:01.908204+0000","last_clean":"2024-04-25T11:45:01.908204+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:45:01.908204+0000","last_undegraded":"2024-04-25T11:45:01.908204+0000","last_fullsized":"2024-04-25T11:45:01.908204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4856","reported_seq":4960,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.868365+0000","last_change":"2024-04-25T11:20:24.096898+0000","last_active":"2024-04-25T11:45:01.868365+0000","last_peered":"2024-04-25T11:45:01.868365+0000","last_clean":"2024-04-25T11:45:01.868365+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:01.868365+0000","last_undegraded":"2024-04-25T11:45:01.868365+0000","last_fullsized":"2024-04-25T11:45:01.868365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:33.109959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20164,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'30","reported_seq":124,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.664575+0000","last_change":"2024-04-25T11:20:19.042825+0000","last_active":"2024-04-25T11:45:01.664575+0000","last_peered":"2024-04-25T11:45:01.664575+0000","last_clean":"2024-04-25T11:45:01.664575+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:45:01.664575+0000","last_undegraded":"2024-04-25T11:45:01.664575+0000","last_fullsized":"2024-04-25T11:45:01.664575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.244899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":54,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:50.299464+0000","last_change":"2024-04-25T11:21:00.073541+0000","last_active":"2024-04-25T11:44:50.299464+0000","last_peered":"2024-04-25T11:44:50.299464+0000","last_clean":"2024-04-25T11:44:50.299464+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:44:50.299464+0000","last_undegraded":"2024-04-25T11:44:50.299464+0000","last_fullsized":"2024-04-25T11:44:50.299464+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4265","reported_seq":4360,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.832608+0000","last_change":"2024-04-25T11:20:24.098698+0000","last_active":"2024-04-25T11:45:01.832608+0000","last_peered":"2024-04-25T11:45:01.832608+0000","last_clean":"2024-04-25T11:45:01.832608+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:01.832608+0000","last_undegraded":"2024-04-25T11:45:01.832608+0000","last_fullsized":"2024-04-25T11:45:01.832608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17222,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"40'8","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.108350+0000","last_change":"2024-04-25T11:20:19.047360+0000","last_active":"2024-04-25T11:44:19.108350+0000","last_peered":"2024-04-25T11:44:19.108350+0000","last_clean":"2024-04-25T11:44:19.108350+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:44:19.108350+0000","last_undegraded":"2024-04-25T11:44:19.108350+0000","last_fullsized":"2024-04-25T11:44:19.108350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":62,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.649419+0000","last_change":"2024-04-25T11:21:00.073623+0000","last_active":"2024-04-25T11:45:01.649419+0000","last_peered":"2024-04-25T11:45:01.649419+0000","last_clean":"2024-04-25T11:45:01.649419+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:01.649419+0000","last_undegraded":"2024-04-25T11:45:01.649419+0000","last_fullsized":"2024-04-25T11:45:01.649419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4169","reported_seq":4272,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.641545+0000","last_change":"2024-04-25T11:20:24.095741+0000","last_active":"2024-04-25T11:45:01.641545+0000","last_peered":"2024-04-25T11:45:01.641545+0000","last_clean":"2024-04-25T11:45:01.641545+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:01.641545+0000","last_undegraded":"2024-04-25T11:45:01.641545+0000","last_fullsized":"2024-04-25T11:45:01.641545+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:53.137795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16708,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'10","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.781078+0000","last_change":"2024-04-25T11:20:20.018772+0000","last_active":"2024-04-25T11:45:01.781078+0000","last_peered":"2024-04-25T11:45:01.781078+0000","last_clean":"2024-04-25T11:45:01.781078+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:45:01.781078+0000","last_undegraded":"2024-04-25T11:45:01.781078+0000","last_fullsized":"2024-04-25T11:45:01.781078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1090","reported_seq":1170,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.759715+0000","last_change":"2024-04-25T11:20:19.043729+0000","last_active":"2024-04-25T11:45:01.759715+0000","last_peered":"2024-04-25T11:45:01.759715+0000","last_clean":"2024-04-25T11:45:01.759715+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:45:01.759715+0000","last_undegraded":"2024-04-25T11:45:01.759715+0000","last_fullsized":"2024-04-25T11:45:01.759715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5159","reported_seq":5257,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.881246+0000","last_change":"2024-04-25T11:20:24.096331+0000","last_active":"2024-04-25T11:45:01.881246+0000","last_peered":"2024-04-25T11:45:01.881246+0000","last_clean":"2024-04-25T11:45:01.881246+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:01.881246+0000","last_undegraded":"2024-04-25T11:45:01.881246+0000","last_fullsized":"2024-04-25T11:45:01.881246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21468,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:03.970063+0000","last_change":"2024-04-25T11:20:20.034793+0000","last_active":"2024-04-25T11:45:03.970063+0000","last_peered":"2024-04-25T11:45:03.970063+0000","last_clean":"2024-04-25T11:45:03.970063+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:45:03.970063+0000","last_undegraded":"2024-04-25T11:45:03.970063+0000","last_fullsized":"2024-04-25T11:45:03.970063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"40'13","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.476916+0000","last_change":"2024-04-25T11:20:19.001291+0000","last_active":"2024-04-25T11:45:01.476916+0000","last_peered":"2024-04-25T11:45:01.476916+0000","last_clean":"2024-04-25T11:45:01.476916+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:45:01.476916+0000","last_undegraded":"2024-04-25T11:45:01.476916+0000","last_fullsized":"2024-04-25T11:45:01.476916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5171","reported_seq":5268,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.911691+0000","last_change":"2024-04-25T11:20:24.100695+0000","last_active":"2024-04-25T11:45:01.911691+0000","last_peered":"2024-04-25T11:45:01.911691+0000","last_clean":"2024-04-25T11:45:01.911691+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:01.911691+0000","last_undegraded":"2024-04-25T11:45:01.911691+0000","last_fullsized":"2024-04-25T11:45:01.911691+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21566,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.795768+0000","last_change":"2024-04-25T11:20:20.034532+0000","last_active":"2024-04-25T11:44:53.795768+0000","last_peered":"2024-04-25T11:44:53.795768+0000","last_clean":"2024-04-25T11:44:53.795768+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:44:53.795768+0000","last_undegraded":"2024-04-25T11:44:53.795768+0000","last_fullsized":"2024-04-25T11:44:53.795768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"40'9","reported_seq":95,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.733767+0000","last_change":"2024-04-25T11:20:19.032459+0000","last_active":"2024-04-25T11:45:01.733767+0000","last_peered":"2024-04-25T11:45:01.733767+0000","last_clean":"2024-04-25T11:45:01.733767+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:45:01.733767+0000","last_undegraded":"2024-04-25T11:45:01.733767+0000","last_fullsized":"2024-04-25T11:45:01.733767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6708,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.767277+0000","last_change":"2024-04-25T11:21:00.113833+0000","last_active":"2024-04-25T11:45:01.767277+0000","last_peered":"2024-04-25T11:45:01.767277+0000","last_clean":"2024-04-25T11:45:01.767277+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:01.767277+0000","last_undegraded":"2024-04-25T11:45:01.767277+0000","last_fullsized":"2024-04-25T11:45:01.767277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:19.233582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.688232+0000","last_change":"2024-04-25T11:20:20.007786+0000","last_active":"2024-04-25T11:44:58.688232+0000","last_peered":"2024-04-25T11:44:58.688232+0000","last_clean":"2024-04-25T11:44:58.688232+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:44:58.688232+0000","last_undegraded":"2024-04-25T11:44:58.688232+0000","last_fullsized":"2024-04-25T11:44:58.688232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:35.103561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'7","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:56.006809+0000","last_change":"2024-04-25T11:20:19.036417+0000","last_active":"2024-04-25T11:44:56.006809+0000","last_peered":"2024-04-25T11:44:56.006809+0000","last_clean":"2024-04-25T11:44:56.006809+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:44:56.006809+0000","last_undegraded":"2024-04-25T11:44:56.006809+0000","last_fullsized":"2024-04-25T11:44:56.006809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:12.823660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4795","reported_seq":4891,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.813188+0000","last_change":"2024-04-25T11:20:24.096497+0000","last_active":"2024-04-25T11:45:01.813188+0000","last_peered":"2024-04-25T11:45:01.813188+0000","last_clean":"2024-04-25T11:45:01.813188+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:01.813188+0000","last_undegraded":"2024-04-25T11:45:01.813188+0000","last_fullsized":"2024-04-25T11:45:01.813188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19675,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'18","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.833423+0000","last_change":"2024-04-25T11:20:20.032289+0000","last_active":"2024-04-25T11:44:57.833423+0000","last_peered":"2024-04-25T11:44:57.833423+0000","last_clean":"2024-04-25T11:44:57.833423+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:44:57.833423+0000","last_undegraded":"2024-04-25T11:44:57.833423+0000","last_fullsized":"2024-04-25T11:44:57.833423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'14","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.809423+0000","last_change":"2024-04-25T11:20:19.039620+0000","last_active":"2024-04-25T11:45:01.809423+0000","last_peered":"2024-04-25T11:45:01.809423+0000","last_clean":"2024-04-25T11:45:01.809423+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:45:01.809423+0000","last_undegraded":"2024-04-25T11:45:01.809423+0000","last_fullsized":"2024-04-25T11:45:01.809423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4482","reported_seq":4583,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.606413+0000","last_change":"2024-04-25T11:20:24.167601+0000","last_active":"2024-04-25T11:45:01.606413+0000","last_peered":"2024-04-25T11:45:01.606413+0000","last_clean":"2024-04-25T11:45:01.606413+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:01.606413+0000","last_undegraded":"2024-04-25T11:45:01.606413+0000","last_fullsized":"2024-04-25T11:45:01.606413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:20.993268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18513,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"40'13","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.888467+0000","last_change":"2024-04-25T11:20:20.036057+0000","last_active":"2024-04-25T11:45:01.888467+0000","last_peered":"2024-04-25T11:45:01.888467+0000","last_clean":"2024-04-25T11:45:01.888467+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:45:01.888467+0000","last_undegraded":"2024-04-25T11:45:01.888467+0000","last_fullsized":"2024-04-25T11:45:01.888467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'369","reported_seq":449,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.741813+0000","last_change":"2024-04-25T11:20:19.035799+0000","last_active":"2024-04-25T11:45:01.741813+0000","last_peered":"2024-04-25T11:45:01.741813+0000","last_clean":"2024-04-25T11:45:01.741813+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:45:01.741813+0000","last_undegraded":"2024-04-25T11:45:01.741813+0000","last_fullsized":"2024-04-25T11:45:01.741813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:56:06.598918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4559","reported_seq":4660,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.922392+0000","last_change":"2024-04-25T11:20:24.162940+0000","last_active":"2024-04-25T11:45:01.922392+0000","last_peered":"2024-04-25T11:45:01.922392+0000","last_clean":"2024-04-25T11:45:01.922392+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:45:01.922392+0000","last_undegraded":"2024-04-25T11:45:01.922392+0000","last_fullsized":"2024-04-25T11:45:01.922392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19035,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'9","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.642438+0000","last_change":"2024-04-25T11:20:20.032139+0000","last_active":"2024-04-25T11:45:01.642438+0000","last_peered":"2024-04-25T11:45:01.642438+0000","last_clean":"2024-04-25T11:45:01.642438+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:45:01.642438+0000","last_undegraded":"2024-04-25T11:45:01.642438+0000","last_fullsized":"2024-04-25T11:45:01.642438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'6","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.106657+0000","last_change":"2024-04-25T11:20:19.042658+0000","last_active":"2024-04-25T11:44:19.106657+0000","last_peered":"2024-04-25T11:44:19.106657+0000","last_clean":"2024-04-25T11:44:19.106657+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:44:19.106657+0000","last_undegraded":"2024-04-25T11:44:19.106657+0000","last_fullsized":"2024-04-25T11:44:19.106657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:41.107653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5085","reported_seq":5186,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.495408+0000","last_change":"2024-04-25T11:20:24.161667+0000","last_active":"2024-04-25T11:45:01.495408+0000","last_peered":"2024-04-25T11:45:01.495408+0000","last_clean":"2024-04-25T11:45:01.495408+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:45:01.495408+0000","last_undegraded":"2024-04-25T11:45:01.495408+0000","last_fullsized":"2024-04-25T11:45:01.495408+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21093,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'14","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.773913+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:45:01.773913+0000","last_peered":"2024-04-25T11:45:01.773913+0000","last_clean":"2024-04-25T11:45:01.773913+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:45:01.773913+0000","last_undegraded":"2024-04-25T11:45:01.773913+0000","last_fullsized":"2024-04-25T11:45:01.773913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"40'5","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.714528+0000","last_change":"2024-04-25T11:20:18.999972+0000","last_active":"2024-04-25T11:45:01.714528+0000","last_peered":"2024-04-25T11:45:01.714528+0000","last_clean":"2024-04-25T11:45:01.714528+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:45:01.714528+0000","last_undegraded":"2024-04-25T11:45:01.714528+0000","last_fullsized":"2024-04-25T11:45:01.714528+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5304","reported_seq":5410,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.825382+0000","last_change":"2024-04-25T11:20:24.093462+0000","last_active":"2024-04-25T11:45:01.825382+0000","last_peered":"2024-04-25T11:45:01.825382+0000","last_clean":"2024-04-25T11:45:01.825382+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:01.825382+0000","last_undegraded":"2024-04-25T11:45:01.825382+0000","last_fullsized":"2024-04-25T11:45:01.825382+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21947,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.886490+0000","last_change":"2024-04-25T11:20:20.013185+0000","last_active":"2024-04-25T11:44:45.886490+0000","last_peered":"2024-04-25T11:44:45.886490+0000","last_clean":"2024-04-25T11:44:45.886490+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:44:45.886490+0000","last_undegraded":"2024-04-25T11:44:45.886490+0000","last_fullsized":"2024-04-25T11:44:45.886490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.837174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'15","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:24.175082+0000","last_change":"2024-04-25T11:20:19.022551+0000","last_active":"2024-04-25T11:44:24.175082+0000","last_peered":"2024-04-25T11:44:24.175082+0000","last_clean":"2024-04-25T11:44:24.175082+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:44:24.175082+0000","last_undegraded":"2024-04-25T11:44:24.175082+0000","last_fullsized":"2024-04-25T11:44:24.175082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5442","reported_seq":5552,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.691672+0000","last_change":"2024-04-25T11:20:24.085764+0000","last_active":"2024-04-25T11:45:01.691672+0000","last_peered":"2024-04-25T11:45:01.691672+0000","last_clean":"2024-04-25T11:45:01.691672+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:01.691672+0000","last_undegraded":"2024-04-25T11:45:01.691672+0000","last_fullsized":"2024-04-25T11:45:01.691672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23011,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"40'12","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.768223+0000","last_change":"2024-04-25T11:20:20.023794+0000","last_active":"2024-04-25T11:45:01.768223+0000","last_peered":"2024-04-25T11:45:01.768223+0000","last_clean":"2024-04-25T11:45:01.768223+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:45:01.768223+0000","last_undegraded":"2024-04-25T11:45:01.768223+0000","last_fullsized":"2024-04-25T11:45:01.768223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'64","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.482378+0000","last_change":"2024-04-25T11:20:19.004104+0000","last_active":"2024-04-25T11:45:01.482378+0000","last_peered":"2024-04-25T11:45:01.482378+0000","last_clean":"2024-04-25T11:45:01.482378+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:01.482378+0000","last_undegraded":"2024-04-25T11:45:01.482378+0000","last_fullsized":"2024-04-25T11:45:01.482378+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5066","reported_seq":5172,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.881554+0000","last_change":"2024-04-25T11:20:24.069072+0000","last_active":"2024-04-25T11:45:01.881554+0000","last_peered":"2024-04-25T11:45:01.881554+0000","last_clean":"2024-04-25T11:45:01.881554+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:01.881554+0000","last_undegraded":"2024-04-25T11:45:01.881554+0000","last_fullsized":"2024-04-25T11:45:01.881554+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21672,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'6","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.416658+0000","last_change":"2024-04-25T11:20:20.023563+0000","last_active":"2024-04-25T11:44:53.416658+0000","last_peered":"2024-04-25T11:44:53.416658+0000","last_clean":"2024-04-25T11:44:53.416658+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:44:53.416658+0000","last_undegraded":"2024-04-25T11:44:53.416658+0000","last_fullsized":"2024-04-25T11:44:53.416658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'18","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.750602+0000","last_change":"2024-04-25T11:20:19.039032+0000","last_active":"2024-04-25T11:45:01.750602+0000","last_peered":"2024-04-25T11:45:01.750602+0000","last_clean":"2024-04-25T11:45:01.750602+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:45:01.750602+0000","last_undegraded":"2024-04-25T11:45:01.750602+0000","last_fullsized":"2024-04-25T11:45:01.750602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5144","reported_seq":5239,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.826280+0000","last_change":"2024-04-25T11:20:24.155893+0000","last_active":"2024-04-25T11:45:01.826280+0000","last_peered":"2024-04-25T11:45:01.826280+0000","last_clean":"2024-04-25T11:45:01.826280+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:01.826280+0000","last_undegraded":"2024-04-25T11:45:01.826280+0000","last_fullsized":"2024-04-25T11:45:01.826280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5144,"log_dups_size":0,"ondisk_log_size":5144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:08.052538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21523,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"40'5","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.873720+0000","last_change":"2024-04-25T11:20:20.048529+0000","last_active":"2024-04-25T11:45:01.873720+0000","last_peered":"2024-04-25T11:45:01.873720+0000","last_clean":"2024-04-25T11:45:01.873720+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:45:01.873720+0000","last_undegraded":"2024-04-25T11:45:01.873720+0000","last_fullsized":"2024-04-25T11:45:01.873720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"40'9","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.490470+0000","last_change":"2024-04-25T11:20:19.009549+0000","last_active":"2024-04-25T11:45:01.490470+0000","last_peered":"2024-04-25T11:45:01.490470+0000","last_clean":"2024-04-25T11:45:01.490470+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:45:01.490470+0000","last_undegraded":"2024-04-25T11:45:01.490470+0000","last_fullsized":"2024-04-25T11:45:01.490470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":162,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.105537+0000","last_change":"2024-04-25T11:20:08.217795+0000","last_active":"2024-04-25T11:44:19.105537+0000","last_peered":"2024-04-25T11:44:19.105537+0000","last_clean":"2024-04-25T11:44:19.105537+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:44:19.105537+0000","last_undegraded":"2024-04-25T11:44:19.105537+0000","last_fullsized":"2024-04-25T11:44:19.105537+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:06.845251+0000","last_clean_scrub_stamp":"2024-04-25T11:20:06.845251+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:09.272427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5040","reported_seq":5135,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.786865+0000","last_change":"2024-04-25T11:20:24.167014+0000","last_active":"2024-04-25T11:45:01.786865+0000","last_peered":"2024-04-25T11:45:01.786865+0000","last_clean":"2024-04-25T11:45:01.786865+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:01.786865+0000","last_undegraded":"2024-04-25T11:45:01.786865+0000","last_fullsized":"2024-04-25T11:45:01.786865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:37.179641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21819,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'14","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.941132+0000","last_change":"2024-04-25T11:20:20.048267+0000","last_active":"2024-04-25T11:45:01.941132+0000","last_peered":"2024-04-25T11:45:01.941132+0000","last_clean":"2024-04-25T11:45:01.941132+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:45:01.941132+0000","last_undegraded":"2024-04-25T11:45:01.941132+0000","last_fullsized":"2024-04-25T11:45:01.941132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'16","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.758201+0000","last_change":"2024-04-25T11:20:19.037300+0000","last_active":"2024-04-25T11:45:01.758201+0000","last_peered":"2024-04-25T11:45:01.758201+0000","last_clean":"2024-04-25T11:45:01.758201+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:01.758201+0000","last_undegraded":"2024-04-25T11:45:01.758201+0000","last_fullsized":"2024-04-25T11:45:01.758201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5278","reported_seq":5376,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.777567+0000","last_change":"2024-04-25T11:20:24.162969+0000","last_active":"2024-04-25T11:45:01.777567+0000","last_peered":"2024-04-25T11:45:01.777567+0000","last_clean":"2024-04-25T11:45:01.777567+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:01.777567+0000","last_undegraded":"2024-04-25T11:45:01.777567+0000","last_fullsized":"2024-04-25T11:45:01.777567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:44.151480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22099,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"40'11","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.935174+0000","last_change":"2024-04-25T11:20:20.046381+0000","last_active":"2024-04-25T11:45:01.935174+0000","last_peered":"2024-04-25T11:45:01.935174+0000","last_clean":"2024-04-25T11:45:01.935174+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:45:01.935174+0000","last_undegraded":"2024-04-25T11:45:01.935174+0000","last_fullsized":"2024-04-25T11:45:01.935174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'14","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.796982+0000","last_change":"2024-04-25T11:20:19.019541+0000","last_active":"2024-04-25T11:45:01.796982+0000","last_peered":"2024-04-25T11:45:01.796982+0000","last_clean":"2024-04-25T11:45:01.796982+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:45:01.796982+0000","last_undegraded":"2024-04-25T11:45:01.796982+0000","last_fullsized":"2024-04-25T11:45:01.796982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:30.109391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4958","reported_seq":5058,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.858968+0000","last_change":"2024-04-25T11:21:00.479066+0000","last_active":"2024-04-25T11:45:01.858968+0000","last_peered":"2024-04-25T11:45:01.858968+0000","last_clean":"2024-04-25T11:45:01.858968+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:01.858968+0000","last_undegraded":"2024-04-25T11:45:01.858968+0000","last_fullsized":"2024-04-25T11:45:01.858968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:14.274164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20958,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:58.040257+0000","last_change":"2024-04-25T11:20:20.051333+0000","last_active":"2024-04-25T11:44:58.040257+0000","last_peered":"2024-04-25T11:44:58.040257+0000","last_clean":"2024-04-25T11:44:58.040257+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:44:58.040257+0000","last_undegraded":"2024-04-25T11:44:58.040257+0000","last_fullsized":"2024-04-25T11:44:58.040257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:51:26.705007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'12","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.741527+0000","last_change":"2024-04-25T11:20:19.005433+0000","last_active":"2024-04-25T11:45:01.741527+0000","last_peered":"2024-04-25T11:45:01.741527+0000","last_clean":"2024-04-25T11:45:01.741527+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:01.741527+0000","last_undegraded":"2024-04-25T11:45:01.741527+0000","last_fullsized":"2024-04-25T11:45:01.741527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4822","reported_seq":4925,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.802783+0000","last_change":"2024-04-25T11:20:24.158620+0000","last_active":"2024-04-25T11:45:01.802783+0000","last_peered":"2024-04-25T11:45:01.802783+0000","last_clean":"2024-04-25T11:45:01.802783+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:01.802783+0000","last_undegraded":"2024-04-25T11:45:01.802783+0000","last_fullsized":"2024-04-25T11:45:01.802783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:33:30.064466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20365,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'8","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.813613+0000","last_change":"2024-04-25T11:20:20.049213+0000","last_active":"2024-04-25T11:45:01.813613+0000","last_peered":"2024-04-25T11:45:01.813613+0000","last_clean":"2024-04-25T11:45:01.813613+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:45:01.813613+0000","last_undegraded":"2024-04-25T11:45:01.813613+0000","last_fullsized":"2024-04-25T11:45:01.813613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'7","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.758060+0000","last_change":"2024-04-25T11:20:19.019786+0000","last_active":"2024-04-25T11:45:01.758060+0000","last_peered":"2024-04-25T11:45:01.758060+0000","last_clean":"2024-04-25T11:45:01.758060+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:45:01.758060+0000","last_undegraded":"2024-04-25T11:45:01.758060+0000","last_fullsized":"2024-04-25T11:45:01.758060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4349","reported_seq":4444,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.652787+0000","last_change":"2024-04-25T11:20:24.100636+0000","last_active":"2024-04-25T11:45:01.652787+0000","last_peered":"2024-04-25T11:45:01.652787+0000","last_clean":"2024-04-25T11:45:01.652787+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:01.652787+0000","last_undegraded":"2024-04-25T11:45:01.652787+0000","last_fullsized":"2024-04-25T11:45:01.652787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:06.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18465,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.554355+0000","last_change":"2024-04-25T11:20:20.042215+0000","last_active":"2024-04-25T11:45:01.554355+0000","last_peered":"2024-04-25T11:45:01.554355+0000","last_clean":"2024-04-25T11:45:01.554355+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:45:01.554355+0000","last_undegraded":"2024-04-25T11:45:01.554355+0000","last_fullsized":"2024-04-25T11:45:01.554355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:33.116153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'31","reported_seq":126,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732025+0000","last_change":"2024-04-25T11:20:19.005148+0000","last_active":"2024-04-25T11:45:01.732025+0000","last_peered":"2024-04-25T11:45:01.732025+0000","last_clean":"2024-04-25T11:45:01.732025+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:01.732025+0000","last_undegraded":"2024-04-25T11:45:01.732025+0000","last_fullsized":"2024-04-25T11:45:01.732025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5276","reported_seq":5374,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.844891+0000","last_change":"2024-04-25T11:20:24.090629+0000","last_active":"2024-04-25T11:45:01.844891+0000","last_peered":"2024-04-25T11:45:01.844891+0000","last_clean":"2024-04-25T11:45:01.844891+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:45:01.844891+0000","last_undegraded":"2024-04-25T11:45:01.844891+0000","last_fullsized":"2024-04-25T11:45:01.844891+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:49.013402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21946,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'8","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.128429+0000","last_change":"2024-04-25T11:20:20.002670+0000","last_active":"2024-04-25T11:44:57.128429+0000","last_peered":"2024-04-25T11:44:57.128429+0000","last_clean":"2024-04-25T11:44:57.128429+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:44:57.128429+0000","last_undegraded":"2024-04-25T11:44:57.128429+0000","last_fullsized":"2024-04-25T11:44:57.128429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'16","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.734297+0000","last_change":"2024-04-25T11:20:19.030108+0000","last_active":"2024-04-25T11:45:01.734297+0000","last_peered":"2024-04-25T11:45:01.734297+0000","last_clean":"2024-04-25T11:45:01.734297+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:45:01.734297+0000","last_undegraded":"2024-04-25T11:45:01.734297+0000","last_fullsized":"2024-04-25T11:45:01.734297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:22.432058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5730","reported_seq":5848,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.957084+0000","last_change":"2024-04-25T11:20:24.155671+0000","last_active":"2024-04-25T11:45:01.957084+0000","last_peered":"2024-04-25T11:45:01.957084+0000","last_clean":"2024-04-25T11:45:01.957084+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:45:01.957084+0000","last_undegraded":"2024-04-25T11:45:01.957084+0000","last_fullsized":"2024-04-25T11:45:01.957084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23847,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"40'15","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.678370+0000","last_change":"2024-04-25T11:20:20.049483+0000","last_active":"2024-04-25T11:45:01.678370+0000","last_peered":"2024-04-25T11:45:01.678370+0000","last_clean":"2024-04-25T11:45:01.678370+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:45:01.678370+0000","last_undegraded":"2024-04-25T11:45:01.678370+0000","last_fullsized":"2024-04-25T11:45:01.678370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:40.087992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"40'11","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.828764+0000","last_change":"2024-04-25T11:20:19.024679+0000","last_active":"2024-04-25T11:45:01.828764+0000","last_peered":"2024-04-25T11:45:01.828764+0000","last_clean":"2024-04-25T11:45:01.828764+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:45:01.828764+0000","last_undegraded":"2024-04-25T11:45:01.828764+0000","last_fullsized":"2024-04-25T11:45:01.828764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:56:40.028795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4272","reported_seq":4381,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.913895+0000","last_change":"2024-04-25T11:20:24.162642+0000","last_active":"2024-04-25T11:45:01.913895+0000","last_peered":"2024-04-25T11:45:01.913895+0000","last_clean":"2024-04-25T11:45:01.913895+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:45:01.913895+0000","last_undegraded":"2024-04-25T11:45:01.913895+0000","last_fullsized":"2024-04-25T11:45:01.913895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17563,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.625058+0000","last_change":"2024-04-25T11:20:20.019811+0000","last_active":"2024-04-25T11:45:01.625058+0000","last_peered":"2024-04-25T11:45:01.625058+0000","last_clean":"2024-04-25T11:45:01.625058+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:45:01.625058+0000","last_undegraded":"2024-04-25T11:45:01.625058+0000","last_fullsized":"2024-04-25T11:45:01.625058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:42.121167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'13","reported_seq":102,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.812188+0000","last_change":"2024-04-25T11:20:19.022522+0000","last_active":"2024-04-25T11:45:01.812188+0000","last_peered":"2024-04-25T11:45:01.812188+0000","last_clean":"2024-04-25T11:45:01.812188+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:45:01.812188+0000","last_undegraded":"2024-04-25T11:45:01.812188+0000","last_fullsized":"2024-04-25T11:45:01.812188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:43.003191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4282","reported_seq":4376,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.271935+0000","last_change":"2024-04-25T11:20:24.158282+0000","last_active":"2024-04-25T11:45:01.271935+0000","last_peered":"2024-04-25T11:45:01.271935+0000","last_clean":"2024-04-25T11:45:01.271935+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:01.271935+0000","last_undegraded":"2024-04-25T11:45:01.271935+0000","last_fullsized":"2024-04-25T11:45:01.271935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:38.975232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17949,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:45.022826+0000","last_change":"2024-04-25T11:20:20.043655+0000","last_active":"2024-04-25T11:44:45.022826+0000","last_peered":"2024-04-25T11:44:45.022826+0000","last_clean":"2024-04-25T11:44:45.022826+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:44:45.022826+0000","last_undegraded":"2024-04-25T11:44:45.022826+0000","last_fullsized":"2024-04-25T11:44:45.022826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:19.905799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'348","reported_seq":441,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:42.649851+0000","last_change":"2024-04-25T11:20:19.027927+0000","last_active":"2024-04-25T11:44:42.649851+0000","last_peered":"2024-04-25T11:44:42.649851+0000","last_clean":"2024-04-25T11:44:42.649851+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:44:42.649851+0000","last_undegraded":"2024-04-25T11:44:42.649851+0000","last_fullsized":"2024-04-25T11:44:42.649851+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4220","reported_seq":4318,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.882292+0000","last_change":"2024-04-25T11:20:24.085271+0000","last_active":"2024-04-25T11:45:01.882292+0000","last_peered":"2024-04-25T11:45:01.882292+0000","last_clean":"2024-04-25T11:45:01.882292+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:01.882292+0000","last_undegraded":"2024-04-25T11:45:01.882292+0000","last_fullsized":"2024-04-25T11:45:01.882292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17734,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.699681+0000","last_change":"2024-04-25T11:20:20.024231+0000","last_active":"2024-04-25T11:45:01.699681+0000","last_peered":"2024-04-25T11:45:01.699681+0000","last_clean":"2024-04-25T11:45:01.699681+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:45:01.699681+0000","last_undegraded":"2024-04-25T11:45:01.699681+0000","last_fullsized":"2024-04-25T11:45:01.699681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'30","reported_seq":105,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.791534+0000","last_change":"2024-04-25T11:20:19.037894+0000","last_active":"2024-04-25T11:45:01.791534+0000","last_peered":"2024-04-25T11:45:01.791534+0000","last_clean":"2024-04-25T11:45:01.791534+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:45:01.791534+0000","last_undegraded":"2024-04-25T11:45:01.791534+0000","last_fullsized":"2024-04-25T11:45:01.791534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:09.828329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":32,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4473","reported_seq":4562,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.717937+0000","last_change":"2024-04-25T11:20:24.096512+0000","last_active":"2024-04-25T11:45:01.717937+0000","last_peered":"2024-04-25T11:45:01.717937+0000","last_clean":"2024-04-25T11:45:01.717937+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:01.717937+0000","last_undegraded":"2024-04-25T11:45:01.717937+0000","last_fullsized":"2024-04-25T11:45:01.717937+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18511,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'14","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.581547+0000","last_change":"2024-04-25T11:20:20.033538+0000","last_active":"2024-04-25T11:45:01.581547+0000","last_peered":"2024-04-25T11:45:01.581547+0000","last_clean":"2024-04-25T11:45:01.581547+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:45:01.581547+0000","last_undegraded":"2024-04-25T11:45:01.581547+0000","last_fullsized":"2024-04-25T11:45:01.581547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:15.362520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"40'7","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.723408+0000","last_change":"2024-04-25T11:20:19.035108+0000","last_active":"2024-04-25T11:45:01.723408+0000","last_peered":"2024-04-25T11:45:01.723408+0000","last_clean":"2024-04-25T11:45:01.723408+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:45:01.723408+0000","last_undegraded":"2024-04-25T11:45:01.723408+0000","last_fullsized":"2024-04-25T11:45:01.723408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:14:34.090047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4342","reported_seq":4446,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.785611+0000","last_change":"2024-04-25T11:21:00.073978+0000","last_active":"2024-04-25T11:45:01.785611+0000","last_peered":"2024-04-25T11:45:01.785611+0000","last_clean":"2024-04-25T11:45:01.785611+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:45:01.785611+0000","last_undegraded":"2024-04-25T11:45:01.785611+0000","last_fullsized":"2024-04-25T11:45:01.785611+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17952,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'15","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.834487+0000","last_change":"2024-04-25T11:20:20.038274+0000","last_active":"2024-04-25T11:45:01.834487+0000","last_peered":"2024-04-25T11:45:01.834487+0000","last_clean":"2024-04-25T11:45:01.834487+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:45:01.834487+0000","last_undegraded":"2024-04-25T11:45:01.834487+0000","last_fullsized":"2024-04-25T11:45:01.834487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:53.759245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'16","reported_seq":91,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.739684+0000","last_change":"2024-04-25T11:20:19.046239+0000","last_active":"2024-04-25T11:45:01.739684+0000","last_peered":"2024-04-25T11:45:01.739684+0000","last_clean":"2024-04-25T11:45:01.739684+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:45:01.739684+0000","last_undegraded":"2024-04-25T11:45:01.739684+0000","last_fullsized":"2024-04-25T11:45:01.739684+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3797","reported_seq":3900,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.946860+0000","last_change":"2024-04-25T11:20:24.162680+0000","last_active":"2024-04-25T11:45:01.946860+0000","last_peered":"2024-04-25T11:45:01.946860+0000","last_clean":"2024-04-25T11:45:01.946860+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:45:01.946860+0000","last_undegraded":"2024-04-25T11:45:01.946860+0000","last_fullsized":"2024-04-25T11:45:01.946860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15973,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5425","reported_seq":5538,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.728941+0000","last_change":"2024-04-25T11:20:24.165952+0000","last_active":"2024-04-25T11:45:01.728941+0000","last_peered":"2024-04-25T11:45:01.728941+0000","last_clean":"2024-04-25T11:45:01.728941+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:45:01.728941+0000","last_undegraded":"2024-04-25T11:45:01.728941+0000","last_fullsized":"2024-04-25T11:45:01.728941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:57.822733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22739,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'9","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.810677+0000","last_change":"2024-04-25T11:20:59.944101+0000","last_active":"2024-04-25T11:45:01.810677+0000","last_peered":"2024-04-25T11:45:01.810677+0000","last_clean":"2024-04-25T11:45:01.810677+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:45:01.810677+0000","last_undegraded":"2024-04-25T11:45:01.810677+0000","last_fullsized":"2024-04-25T11:45:01.810677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:06.631176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'12","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.021180+0000","last_change":"2024-04-25T11:20:19.022758+0000","last_active":"2024-04-25T11:44:31.021180+0000","last_peered":"2024-04-25T11:44:31.021180+0000","last_clean":"2024-04-25T11:44:31.021180+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:44:31.021180+0000","last_undegraded":"2024-04-25T11:44:31.021180+0000","last_fullsized":"2024-04-25T11:44:31.021180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:24:32.750739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5679","reported_seq":5794,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.918478+0000","last_change":"2024-04-25T11:20:24.087565+0000","last_active":"2024-04-25T11:45:01.918478+0000","last_peered":"2024-04-25T11:45:01.918478+0000","last_clean":"2024-04-25T11:45:01.918478+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:01.918478+0000","last_undegraded":"2024-04-25T11:45:01.918478+0000","last_fullsized":"2024-04-25T11:45:01.918478+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:22:23.969287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24249,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'17","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.679314+0000","last_change":"2024-04-25T11:45:08.679340+0000","last_active":"2024-04-25T11:45:08.679314+0000","last_peered":"2024-04-25T11:45:08.679314+0000","last_clean":"2024-04-25T11:45:08.679314+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:45:08.679314+0000","last_undegraded":"2024-04-25T11:45:08.679314+0000","last_fullsized":"2024-04-25T11:45:08.679314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.679230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:24.690578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"40'6","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.817308+0000","last_change":"2024-04-25T11:20:19.028918+0000","last_active":"2024-04-25T11:45:01.817308+0000","last_peered":"2024-04-25T11:45:01.817308+0000","last_clean":"2024-04-25T11:45:01.817308+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:45:01.817308+0000","last_undegraded":"2024-04-25T11:45:01.817308+0000","last_fullsized":"2024-04-25T11:45:01.817308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"40'15","reported_seq":90,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.811529+0000","last_change":"2024-04-25T11:20:19.030355+0000","last_active":"2024-04-25T11:45:01.811529+0000","last_peered":"2024-04-25T11:45:01.811529+0000","last_clean":"2024-04-25T11:45:01.811529+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:45:01.811529+0000","last_undegraded":"2024-04-25T11:45:01.811529+0000","last_fullsized":"2024-04-25T11:45:01.811529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:54.537098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'11","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.793967+0000","last_change":"2024-04-25T11:20:20.023218+0000","last_active":"2024-04-25T11:44:59.793967+0000","last_peered":"2024-04-25T11:44:59.793967+0000","last_clean":"2024-04-25T11:44:59.793967+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:44:59.793967+0000","last_undegraded":"2024-04-25T11:44:59.793967+0000","last_fullsized":"2024-04-25T11:44:59.793967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.368416+0000","last_change":"2024-04-25T11:20:24.081844+0000","last_active":"2024-04-25T11:45:01.368416+0000","last_peered":"2024-04-25T11:45:01.368416+0000","last_clean":"2024-04-25T11:45:01.368416+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:01.368416+0000","last_undegraded":"2024-04-25T11:45:01.368416+0000","last_fullsized":"2024-04-25T11:45:01.368416+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:10:46.184656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'640","reported_seq":715,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.714397+0000","last_change":"2024-04-25T11:20:19.020571+0000","last_active":"2024-04-25T11:45:01.714397+0000","last_peered":"2024-04-25T11:45:01.714397+0000","last_clean":"2024-04-25T11:45:01.714397+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:45:01.714397+0000","last_undegraded":"2024-04-25T11:45:01.714397+0000","last_fullsized":"2024-04-25T11:45:01.714397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:17.959056+0000","last_clean_scrub_stamp":"2024-04-25T11:20:17.959056+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'11","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:53.555638+0000","last_change":"2024-04-25T11:20:20.035251+0000","last_active":"2024-04-25T11:44:53.555638+0000","last_peered":"2024-04-25T11:44:53.555638+0000","last_clean":"2024-04-25T11:44:53.555638+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:44:53.555638+0000","last_undegraded":"2024-04-25T11:44:53.555638+0000","last_fullsized":"2024-04-25T11:44:53.555638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:55.550612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5271","reported_seq":5375,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.834108+0000","last_change":"2024-04-25T11:20:24.090019+0000","last_active":"2024-04-25T11:45:01.834108+0000","last_peered":"2024-04-25T11:45:01.834108+0000","last_clean":"2024-04-25T11:45:01.834108+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:45:01.834108+0000","last_undegraded":"2024-04-25T11:45:01.834108+0000","last_fullsized":"2024-04-25T11:45:01.834108+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:23.036058+0000","last_clean_scrub_stamp":"2024-04-25T11:20:23.036058+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22092,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'15","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.557417+0000","last_change":"2024-04-25T11:20:20.006477+0000","last_active":"2024-04-25T11:45:01.557417+0000","last_peered":"2024-04-25T11:45:01.557417+0000","last_clean":"2024-04-25T11:45:01.557417+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:45:01.557417+0000","last_undegraded":"2024-04-25T11:45:01.557417+0000","last_fullsized":"2024-04-25T11:45:01.557417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.606903+0000","last_change":"2024-04-25T11:20:20.032602+0000","last_active":"2024-04-25T11:45:01.606903+0000","last_peered":"2024-04-25T11:45:01.606903+0000","last_clean":"2024-04-25T11:45:01.606903+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:01.606903+0000","last_undegraded":"2024-04-25T11:45:01.606903+0000","last_fullsized":"2024-04-25T11:45:01.606903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'18","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.730633+0000","last_change":"2024-04-25T11:20:20.046943+0000","last_active":"2024-04-25T11:45:01.730633+0000","last_peered":"2024-04-25T11:45:01.730633+0000","last_clean":"2024-04-25T11:45:01.730633+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:45:01.730633+0000","last_undegraded":"2024-04-25T11:45:01.730633+0000","last_fullsized":"2024-04-25T11:45:01.730633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'9","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.484281+0000","last_change":"2024-04-25T11:20:20.038768+0000","last_active":"2024-04-25T11:45:01.484281+0000","last_peered":"2024-04-25T11:45:01.484281+0000","last_clean":"2024-04-25T11:45:01.484281+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:45:01.484281+0000","last_undegraded":"2024-04-25T11:45:01.484281+0000","last_fullsized":"2024-04-25T11:45:01.484281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:33:15.394253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'10","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.855386+0000","last_change":"2024-04-25T11:20:20.037031+0000","last_active":"2024-04-25T11:45:01.855386+0000","last_peered":"2024-04-25T11:45:01.855386+0000","last_clean":"2024-04-25T11:45:01.855386+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:45:01.855386+0000","last_undegraded":"2024-04-25T11:45:01.855386+0000","last_fullsized":"2024-04-25T11:45:01.855386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'13","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.713136+0000","last_change":"2024-04-25T11:20:20.037671+0000","last_active":"2024-04-25T11:45:01.713136+0000","last_peered":"2024-04-25T11:45:01.713136+0000","last_clean":"2024-04-25T11:45:01.713136+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:01.713136+0000","last_undegraded":"2024-04-25T11:45:01.713136+0000","last_fullsized":"2024-04-25T11:45:01.713136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:18.928984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.739378+0000","last_change":"2024-04-25T11:20:20.018861+0000","last_active":"2024-04-25T11:45:01.739378+0000","last_peered":"2024-04-25T11:45:01.739378+0000","last_clean":"2024-04-25T11:45:01.739378+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:45:01.739378+0000","last_undegraded":"2024-04-25T11:45:01.739378+0000","last_fullsized":"2024-04-25T11:45:01.739378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.806011+0000","last_change":"2024-04-25T11:20:20.033564+0000","last_active":"2024-04-25T11:45:01.806011+0000","last_peered":"2024-04-25T11:45:01.806011+0000","last_clean":"2024-04-25T11:45:01.806011+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:45:01.806011+0000","last_undegraded":"2024-04-25T11:45:01.806011+0000","last_fullsized":"2024-04-25T11:45:01.806011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'12","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.802127+0000","last_change":"2024-04-25T11:20:20.045929+0000","last_active":"2024-04-25T11:45:01.802127+0000","last_peered":"2024-04-25T11:45:01.802127+0000","last_clean":"2024-04-25T11:45:01.802127+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:45:01.802127+0000","last_undegraded":"2024-04-25T11:45:01.802127+0000","last_fullsized":"2024-04-25T11:45:01.802127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'15","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:03.970003+0000","last_change":"2024-04-25T11:20:20.036355+0000","last_active":"2024-04-25T11:45:03.970003+0000","last_peered":"2024-04-25T11:45:03.970003+0000","last_clean":"2024-04-25T11:45:03.970003+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:45:03.970003+0000","last_undegraded":"2024-04-25T11:45:03.970003+0000","last_fullsized":"2024-04-25T11:45:03.970003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'11","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.721245+0000","last_change":"2024-04-25T11:20:20.038643+0000","last_active":"2024-04-25T11:45:01.721245+0000","last_peered":"2024-04-25T11:45:01.721245+0000","last_clean":"2024-04-25T11:45:01.721245+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:45:01.721245+0000","last_undegraded":"2024-04-25T11:45:01.721245+0000","last_fullsized":"2024-04-25T11:45:01.721245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:02.311933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'17","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.756766+0000","last_change":"2024-04-25T11:20:20.017682+0000","last_active":"2024-04-25T11:45:01.756766+0000","last_peered":"2024-04-25T11:45:01.756766+0000","last_clean":"2024-04-25T11:45:01.756766+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:45:01.756766+0000","last_undegraded":"2024-04-25T11:45:01.756766+0000","last_fullsized":"2024-04-25T11:45:01.756766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:30.494168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'11","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.761964+0000","last_change":"2024-04-25T11:20:20.018453+0000","last_active":"2024-04-25T11:45:01.761964+0000","last_peered":"2024-04-25T11:45:01.761964+0000","last_clean":"2024-04-25T11:45:01.761964+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:45:01.761964+0000","last_undegraded":"2024-04-25T11:45:01.761964+0000","last_fullsized":"2024-04-25T11:45:01.761964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:20:49.905271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'13","reported_seq":61,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:57.542667+0000","last_change":"2024-04-25T11:20:59.942848+0000","last_active":"2024-04-25T11:44:57.542667+0000","last_peered":"2024-04-25T11:44:57.542667+0000","last_clean":"2024-04-25T11:44:57.542667+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:44:57.542667+0000","last_undegraded":"2024-04-25T11:44:57.542667+0000","last_fullsized":"2024-04-25T11:44:57.542667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:10.210675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.732164+0000","last_change":"2024-04-25T11:20:20.024011+0000","last_active":"2024-04-25T11:45:01.732164+0000","last_peered":"2024-04-25T11:45:01.732164+0000","last_clean":"2024-04-25T11:45:01.732164+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:01.732164+0000","last_undegraded":"2024-04-25T11:45:01.732164+0000","last_fullsized":"2024-04-25T11:45:01.732164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:31.122905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.755387+0000","last_change":"2024-04-25T11:20:20.040819+0000","last_active":"2024-04-25T11:45:01.755387+0000","last_peered":"2024-04-25T11:45:01.755387+0000","last_clean":"2024-04-25T11:45:01.755387+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:45:01.755387+0000","last_undegraded":"2024-04-25T11:45:01.755387+0000","last_fullsized":"2024-04-25T11:45:01.755387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:36:03.989050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.628994+0000","last_change":"2024-04-25T11:20:20.040338+0000","last_active":"2024-04-25T11:45:01.628994+0000","last_peered":"2024-04-25T11:45:01.628994+0000","last_clean":"2024-04-25T11:45:01.628994+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:45:01.628994+0000","last_undegraded":"2024-04-25T11:45:01.628994+0000","last_fullsized":"2024-04-25T11:45:01.628994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:15.170093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'14","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.492272+0000","last_change":"2024-04-25T11:20:20.026967+0000","last_active":"2024-04-25T11:45:01.492272+0000","last_peered":"2024-04-25T11:45:01.492272+0000","last_clean":"2024-04-25T11:45:01.492272+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:45:01.492272+0000","last_undegraded":"2024-04-25T11:45:01.492272+0000","last_fullsized":"2024-04-25T11:45:01.492272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'16","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.866952+0000","last_change":"2024-04-25T11:20:20.026672+0000","last_active":"2024-04-25T11:45:01.866952+0000","last_peered":"2024-04-25T11:45:01.866952+0000","last_clean":"2024-04-25T11:45:01.866952+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:45:01.866952+0000","last_undegraded":"2024-04-25T11:45:01.866952+0000","last_fullsized":"2024-04-25T11:45:01.866952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:31.258413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'15","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.713511+0000","last_change":"2024-04-25T11:20:20.038119+0000","last_active":"2024-04-25T11:45:01.713511+0000","last_peered":"2024-04-25T11:45:01.713511+0000","last_clean":"2024-04-25T11:45:01.713511+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:45:01.713511+0000","last_undegraded":"2024-04-25T11:45:01.713511+0000","last_fullsized":"2024-04-25T11:45:01.713511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'10","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.617668+0000","last_change":"2024-04-25T11:20:20.008756+0000","last_active":"2024-04-25T11:45:01.617668+0000","last_peered":"2024-04-25T11:45:01.617668+0000","last_clean":"2024-04-25T11:45:01.617668+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:45:01.617668+0000","last_undegraded":"2024-04-25T11:45:01.617668+0000","last_fullsized":"2024-04-25T11:45:01.617668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:10.841843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:31.137154+0000","last_change":"2024-04-25T11:20:20.031997+0000","last_active":"2024-04-25T11:44:31.137154+0000","last_peered":"2024-04-25T11:44:31.137154+0000","last_clean":"2024-04-25T11:44:31.137154+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:44:31.137154+0000","last_undegraded":"2024-04-25T11:44:31.137154+0000","last_fullsized":"2024-04-25T11:44:31.137154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'12","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.479416+0000","last_change":"2024-04-25T11:20:20.041522+0000","last_active":"2024-04-25T11:45:01.479416+0000","last_peered":"2024-04-25T11:45:01.479416+0000","last_clean":"2024-04-25T11:45:01.479416+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:45:01.479416+0000","last_undegraded":"2024-04-25T11:45:01.479416+0000","last_fullsized":"2024-04-25T11:45:01.479416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:08.348780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'15","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:59.158664+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:44:59.158664+0000","last_peered":"2024-04-25T11:44:59.158664+0000","last_clean":"2024-04-25T11:44:59.158664+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:44:59.158664+0000","last_undegraded":"2024-04-25T11:44:59.158664+0000","last_fullsized":"2024-04-25T11:44:59.158664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:06:49.422936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'6","reported_seq":67,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.698494+0000","last_change":"2024-04-25T11:20:20.048715+0000","last_active":"2024-04-25T11:45:01.698494+0000","last_peered":"2024-04-25T11:45:01.698494+0000","last_clean":"2024-04-25T11:45:01.698494+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:45:01.698494+0000","last_undegraded":"2024-04-25T11:45:01.698494+0000","last_fullsized":"2024-04-25T11:45:01.698494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'8","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.850540+0000","last_change":"2024-04-25T11:20:20.037402+0000","last_active":"2024-04-25T11:45:01.850540+0000","last_peered":"2024-04-25T11:45:01.850540+0000","last_clean":"2024-04-25T11:45:01.850540+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:01.850540+0000","last_undegraded":"2024-04-25T11:45:01.850540+0000","last_fullsized":"2024-04-25T11:45:01.850540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:44.949914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'8","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:19.088684+0000","last_change":"2024-04-25T11:20:20.046161+0000","last_active":"2024-04-25T11:44:19.088684+0000","last_peered":"2024-04-25T11:44:19.088684+0000","last_clean":"2024-04-25T11:44:19.088684+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:44:19.088684+0000","last_undegraded":"2024-04-25T11:44:19.088684+0000","last_fullsized":"2024-04-25T11:44:19.088684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:49.074306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'11","reported_seq":72,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.829488+0000","last_change":"2024-04-25T11:20:20.038421+0000","last_active":"2024-04-25T11:45:01.829488+0000","last_peered":"2024-04-25T11:45:01.829488+0000","last_clean":"2024-04-25T11:45:01.829488+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:45:01.829488+0000","last_undegraded":"2024-04-25T11:45:01.829488+0000","last_fullsized":"2024-04-25T11:45:01.829488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:46:35.543329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'5","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:44:54.680131+0000","last_change":"2024-04-25T11:20:20.047845+0000","last_active":"2024-04-25T11:44:54.680131+0000","last_peered":"2024-04-25T11:44:54.680131+0000","last_clean":"2024-04-25T11:44:54.680131+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:44:54.680131+0000","last_undegraded":"2024-04-25T11:44:54.680131+0000","last_fullsized":"2024-04-25T11:44:54.680131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:16.929780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'13","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.747397+0000","last_change":"2024-04-25T11:20:20.014022+0000","last_active":"2024-04-25T11:45:01.747397+0000","last_peered":"2024-04-25T11:45:01.747397+0000","last_clean":"2024-04-25T11:45:01.747397+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:45:01.747397+0000","last_undegraded":"2024-04-25T11:45:01.747397+0000","last_fullsized":"2024-04-25T11:45:01.747397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:03.827016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'6","reported_seq":69,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.728358+0000","last_change":"2024-04-25T11:20:20.029825+0000","last_active":"2024-04-25T11:45:01.728358+0000","last_peered":"2024-04-25T11:45:01.728358+0000","last_clean":"2024-04-25T11:45:01.728358+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:45:01.728358+0000","last_undegraded":"2024-04-25T11:45:01.728358+0000","last_fullsized":"2024-04-25T11:45:01.728358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:29.046471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'7","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.871757+0000","last_change":"2024-04-25T11:20:20.038041+0000","last_active":"2024-04-25T11:45:01.871757+0000","last_peered":"2024-04-25T11:45:01.871757+0000","last_clean":"2024-04-25T11:45:01.871757+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:45:01.871757+0000","last_undegraded":"2024-04-25T11:45:01.871757+0000","last_fullsized":"2024-04-25T11:45:01.871757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:04.635289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654935,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156929,"ondisk_log_size":156929,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3396,"num_write_kb":16253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3139,"ondisk_log_size":3139,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575151,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000184,"kb_used_data":11824,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93371656,"statfs":{"total":96636764160,"available":95612575744,"internally_reserved":0,"allocated":12107776,"data_stored":11840562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1181784,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":93190056,"statfs":{"total":96636764160,"available":95426617344,"internally_reserved":0,"allocated":4669440,"data_stored":4418265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575152,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776304,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93595536,"statfs":{"total":96636764160,"available":95841828864,"internally_reserved":0,"allocated":499712,"data_stored":226431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1131992,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93239848,"statfs":{"total":96636764160,"available":95477604352,"internally_reserved":0,"allocated":4669440,"data_stored":4418133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575149,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465336,"kb_used_data":8752,"kb_used_omap":1,"kb_used_meta":1456574,"kb_avail":92906504,"statfs":{"total":96636764160,"available":95136260096,"internally_reserved":0,"allocated":8962048,"data_stored":8703072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1491532231},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575150,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857148,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":856574,"kb_avail":93514692,"statfs":{"total":96636764160,"available":95759044608,"internally_reserved":0,"allocated":577536,"data_stored":314314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877132229},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575149,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484956,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":1484478,"kb_avail":92886884,"statfs":{"total":96636764160,"available":95116169216,"internally_reserved":0,"allocated":479232,"data_stored":223217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1520105926},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575150,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054448,"kb_used_data":3624,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93317392,"statfs":{"total":96636764160,"available":95557009408,"internally_reserved":0,"allocated":3710976,"data_stored":3449798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:10.427 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.427 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.427 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.427 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.428 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.429 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.430 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.431 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T11:20:06.845251+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=6, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.432 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.433 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.434 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.435 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.436 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-25T11:20:17.959056+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-25T11:20:23.036058+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=23, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.437 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.438 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.439 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:10.440 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T11:45:30.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:30.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 Processor -- start 2024-04-25T11:45:30.592 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 -- start start 2024-04-25T11:45:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff42c0b5060 0x7ff42c0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff42c0ca290 0x7ff42c0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42c15d750 con 0x7ff42c15a430 2024-04-25T11:45:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7ff42c15da30 con 0x7ff42c0ca290 2024-04-25T11:45:30.594 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff430c81640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff42c15dd10 con 0x7ff42c0b5060 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff42c0ca290 0x7ff42c0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff42ad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff42c15a430 0x7ff42c15a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:56012/0 (socket says 172.21.15.136:56012) 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff42ad9d640 1 -- 172.21.15.136:0/2444157302 learned_addr learned my addr 172.21.15.136:0/2444157302 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff42c0ca290 0x7ff42c0ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46880/0 (socket says 172.21.15.136:46880) 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff42a59c640 1 --2- 172.21.15.136:0/2444157302 >> v2:172.21.15.157:3300/0 conn(0x7ff42c0b5060 0x7ff42c0c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 -- 172.21.15.136:0/2444157302 >> v2:172.21.15.157:3300/0 conn(0x7ff42c0b5060 msgr2=0x7ff42c0c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 --2- 172.21.15.136:0/2444157302 >> v2:172.21.15.157:3300/0 conn(0x7ff42c0b5060 0x7ff42c0c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 -- 172.21.15.136:0/2444157302 >> v1:172.21.15.136:6789/0 conn(0x7ff42c15a430 legacy=0x7ff42c15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.595 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.607+0000 7ff429d9b640 1 -- 172.21.15.136:0/2444157302 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c15dff0 con 0x7ff42c0ca290 2024-04-25T11:45:30.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff429d9b640 1 --2- 172.21.15.136:0/2444157302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c0ca290 0x7ff42c0ca6a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff42c068ff0 tx=0x7ff41c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1429228be091f6cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:30.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff42959a640 1 -- 172.21.15.136:0/2444157302 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0b5070 con 0x7ff42c0ca290 2024-04-25T11:45:30.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff42959a640 1 -- 172.21.15.136:0/2444157302 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff41c0a8970 con 0x7ff42c0ca290 2024-04-25T11:45:30.596 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff42959a640 1 -- 172.21.15.136:0/2444157302 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0a8d40 con 0x7ff42c0ca290 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 -- 172.21.15.136:0/2444157302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c0ca290 msgr2=0x7ff42c0ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 --2- 172.21.15.136:0/2444157302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c0ca290 0x7ff42c0ca6a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff42c068ff0 tx=0x7ff41c01ec70 comp rx=0 tx=0).stop 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 -- 172.21.15.136:0/2444157302 shutdown_connections 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 --2- 172.21.15.136:0/2444157302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c0ca290 0x7ff42c0ca6a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 --2- 172.21.15.136:0/2444157302 >> v2:172.21.15.157:3300/0 conn(0x7ff42c0b5060 0x7ff42c0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 -- 172.21.15.136:0/2444157302 >> 172.21.15.136:0/2444157302 conn(0x7ff42c070f70 msgr2=0x7ff42c068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 -- 172.21.15.136:0/2444157302 shutdown_connections 2024-04-25T11:45:30.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.611+0000 7ff430c81640 1 -- 172.21.15.136:0/2444157302 wait complete. 2024-04-25T11:45:30.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 Processor -- start 2024-04-25T11:45:30.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- start start 2024-04-25T11:45:30.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff42c0b5060 0x7ff42c152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 0x7ff42c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c15d3f0 con 0x7ff42c15a430 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c159f10 con 0x7ff42c0ca290 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff42c0c4a60 con 0x7ff42c0b5060 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46884/0 (socket says 172.21.15.136:46884) 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 -- 172.21.15.136:0/3989917949 learned_addr learned my addr 172.21.15.136:0/3989917949 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42a59c640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff42c0b5060 0x7ff42c152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff429d9b640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 0x7ff42c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 -- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff42c0b5060 msgr2=0x7ff42c152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff42c0b5060 0x7ff42c152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 -- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 msgr2=0x7ff42c0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 0x7ff42c0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 -- 172.21.15.136:0/3989917949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c1f4680 con 0x7ff42c15a430 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff429d9b640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 0x7ff42c0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff42ad9d640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff42400c3c0 tx=0x7ff42400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424017070 con 0x7ff42c15a430 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff424005cf0 con 0x7ff42c15a430 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42c1e6590 con 0x7ff42c15a430 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240072d0 con 0x7ff42c15a430 2024-04-25T11:45:30.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42c1e6b50 con 0x7ff42c15a430 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.615+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff42c06a350 con 0x7ff42c15a430 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.619+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff42401a070 con 0x7ff42c15a430 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.619+0000 7ff422ffd640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 0x7ff40c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.619+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7ff42408ece0 con 0x7ff42c15a430 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.619+0000 7ff42a59c640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 0x7ff40c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:30.604 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.619+0000 7ff42a59c640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 0x7ff40c071910 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4140070b0 tx=0x7ff414007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:30.611 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.627+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff42405ad50 con 0x7ff42c15a430 2024-04-25T11:45:30.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.815+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff42c0cadd0 con 0x7ff40c06f400 2024-04-25T11:45:30.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.839+0000 7ff422ffd640 1 -- 172.21.15.136:0/3989917949 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334605 (secure 0 0 0) 0x7ff42c0cadd0 con 0x7ff40c06f400 2024-04-25T11:45:30.826 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:30.828 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 msgr2=0x7ff40c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 0x7ff40c071910 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4140070b0 tx=0x7ff414007040 comp rx=0 tx=0).stop 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 msgr2=0x7ff42c1f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff42400c3c0 tx=0x7ff42400c880 comp rx=0 tx=0).stop 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 shutdown_connections 2024-04-25T11:45:30.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff42c15a430 0x7ff42c1f4140 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff42c0ca290 0x7ff42c0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7ff40c06f400 0x7ff40c071910 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 --2- 172.21.15.136:0/3989917949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff42c0b5060 0x7ff42c152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 >> 172.21.15.136:0/3989917949 conn(0x7ff42c070f70 msgr2=0x7ff42c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 shutdown_connections 2024-04-25T11:45:30.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:30.843+0000 7ff430c81640 1 -- 172.21.15.136:0/3989917949 wait complete. 2024-04-25T11:45:30.847 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2024-04-25T11:45:29.428831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15910156,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658432,"num_write_kb":15736463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160971,"ondisk_log_size":160971,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8952152,"kb_used_data":34840,"kb_used_omap":12,"kb_used_meta":8917235,"kb_avail":746022568,"statfs":{"total":773094113280,"available":763927109632,"internally_reserved":0,"allocated":35676160,"data_stored":33593792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9131249202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006426"},"pg_stats":[{"pgid":"2.1f","version":"40'283","reported_seq":379,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.960822+0000","last_change":"2024-04-25T11:45:08.960822+0000","last_active":"2024-04-25T11:45:08.960822+0000","last_peered":"2024-04-25T11:45:08.960822+0000","last_clean":"2024-04-25T11:45:08.960822+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:45:08.960822+0000","last_undegraded":"2024-04-25T11:45:08.960822+0000","last_fullsized":"2024-04-25T11:45:08.960822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'283","last_scrub_stamp":"2024-04-25T11:45:08.960541+0000","last_deep_scrub":"40'283","last_deep_scrub_stamp":"2024-04-25T11:45:08.960541+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.960541+0000","objects_scrubbed":7,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:28.785626+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'7","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.851320+0000","last_change":"2024-04-25T11:45:16.851449+0000","last_active":"2024-04-25T11:45:16.851320+0000","last_peered":"2024-04-25T11:45:16.851320+0000","last_clean":"2024-04-25T11:45:16.851320+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:45:16.851320+0000","last_undegraded":"2024-04-25T11:45:16.851320+0000","last_fullsized":"2024-04-25T11:45:16.851320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:16.851008+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:16.851008+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.851008+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:27.594650+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5345","reported_seq":5459,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:27.899345+0000","last_change":"2024-04-25T11:45:27.899377+0000","last_active":"2024-04-25T11:45:27.899345+0000","last_peered":"2024-04-25T11:45:27.899345+0000","last_clean":"2024-04-25T11:45:27.899345+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:45:27.899345+0000","last_undegraded":"2024-04-25T11:45:27.899345+0000","last_fullsized":"2024-04-25T11:45:27.899345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5345","last_scrub_stamp":"2024-04-25T11:45:27.899232+0000","last_deep_scrub":"40'5345","last_deep_scrub_stamp":"2024-04-25T11:45:27.899232+0000","last_clean_scrub_stamp":"2024-04-25T11:45:27.899232+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:39.688345+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22060,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'11","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.792774+0000","last_change":"2024-04-25T11:45:13.792841+0000","last_active":"2024-04-25T11:45:13.792774+0000","last_peered":"2024-04-25T11:45:13.792774+0000","last_clean":"2024-04-25T11:45:13.792774+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:45:13.792774+0000","last_undegraded":"2024-04-25T11:45:13.792774+0000","last_fullsized":"2024-04-25T11:45:13.792774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.792577+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.792577+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.792577+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:32.569421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.237635+0000","last_change":"2024-04-25T11:45:17.237711+0000","last_active":"2024-04-25T11:45:17.237635+0000","last_peered":"2024-04-25T11:45:17.237635+0000","last_clean":"2024-04-25T11:45:17.237635+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:45:17.237635+0000","last_undegraded":"2024-04-25T11:45:17.237635+0000","last_fullsized":"2024-04-25T11:45:17.237635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:17.237394+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:17.237394+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.237394+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:36.014238+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4856","reported_seq":4969,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.440747+0000","last_change":"2024-04-25T11:45:19.440776+0000","last_active":"2024-04-25T11:45:19.440747+0000","last_peered":"2024-04-25T11:45:19.440747+0000","last_clean":"2024-04-25T11:45:19.440747+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:19.440747+0000","last_undegraded":"2024-04-25T11:45:19.440747+0000","last_fullsized":"2024-04-25T11:45:19.440747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4856","last_scrub_stamp":"2024-04-25T11:45:19.440645+0000","last_deep_scrub":"40'4856","last_deep_scrub_stamp":"2024-04-25T11:45:19.440645+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.440645+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:28.993315+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20164,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'37","reported_seq":141,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.702835+0000","last_change":"2024-04-25T11:45:15.702835+0000","last_active":"2024-04-25T11:45:15.702835+0000","last_peered":"2024-04-25T11:45:15.702835+0000","last_clean":"2024-04-25T11:45:15.702835+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:45:15.702835+0000","last_undegraded":"2024-04-25T11:45:15.702835+0000","last_fullsized":"2024-04-25T11:45:15.702835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'37","last_scrub_stamp":"2024-04-25T11:45:15.702508+0000","last_deep_scrub":"40'37","last_deep_scrub_stamp":"2024-04-25T11:45:15.702508+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.702508+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:37.392868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":63,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.636572+0000","last_change":"2024-04-25T11:45:14.636646+0000","last_active":"2024-04-25T11:45:14.636572+0000","last_peered":"2024-04-25T11:45:14.636572+0000","last_clean":"2024-04-25T11:45:14.636572+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:45:14.636572+0000","last_undegraded":"2024-04-25T11:45:14.636572+0000","last_fullsized":"2024-04-25T11:45:14.636572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:14.636335+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:14.636335+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.636335+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:40.225075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4265","reported_seq":4370,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.354003+0000","last_change":"2024-04-25T11:45:20.354003+0000","last_active":"2024-04-25T11:45:20.354003+0000","last_peered":"2024-04-25T11:45:20.354003+0000","last_clean":"2024-04-25T11:45:20.354003+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:20.354003+0000","last_undegraded":"2024-04-25T11:45:20.354003+0000","last_fullsized":"2024-04-25T11:45:20.354003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4265","last_scrub_stamp":"2024-04-25T11:45:20.353862+0000","last_deep_scrub":"40'4265","last_deep_scrub_stamp":"2024-04-25T11:45:20.353862+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.353862+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:39.130706+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17222,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"40'11","reported_seq":102,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.736889+0000","last_change":"2024-04-25T11:45:17.736889+0000","last_active":"2024-04-25T11:45:17.736889+0000","last_peered":"2024-04-25T11:45:17.736889+0000","last_clean":"2024-04-25T11:45:17.736889+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:45:17.736889+0000","last_undegraded":"2024-04-25T11:45:17.736889+0000","last_fullsized":"2024-04-25T11:45:17.736889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:17.736473+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:17.736473+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.736473+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:53.767593+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.631676+0000","last_change":"2024-04-25T11:45:15.631751+0000","last_active":"2024-04-25T11:45:15.631676+0000","last_peered":"2024-04-25T11:45:15.631676+0000","last_clean":"2024-04-25T11:45:15.631676+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:15.631676+0000","last_undegraded":"2024-04-25T11:45:15.631676+0000","last_fullsized":"2024-04-25T11:45:15.631676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:15.631432+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:15.631432+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.631432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:57.854886+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4169","reported_seq":4282,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.500583+0000","last_change":"2024-04-25T11:45:15.500583+0000","last_active":"2024-04-25T11:45:15.500583+0000","last_peered":"2024-04-25T11:45:15.500583+0000","last_clean":"2024-04-25T11:45:15.500583+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:15.500583+0000","last_undegraded":"2024-04-25T11:45:15.500583+0000","last_fullsized":"2024-04-25T11:45:15.500583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4169","last_scrub_stamp":"2024-04-25T11:45:15.500247+0000","last_deep_scrub":"40'4169","last_deep_scrub_stamp":"2024-04-25T11:45:15.500247+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.500247+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.015266+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16708,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'10","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.553565+0000","last_change":"2024-04-25T11:45:18.553627+0000","last_active":"2024-04-25T11:45:18.553565+0000","last_peered":"2024-04-25T11:45:18.553565+0000","last_clean":"2024-04-25T11:45:18.553565+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:45:18.553565+0000","last_undegraded":"2024-04-25T11:45:18.553565+0000","last_fullsized":"2024-04-25T11:45:18.553565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:18.553383+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:18.553383+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.553383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:41.251180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1093","reported_seq":1182,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.734685+0000","last_change":"2024-04-25T11:45:16.734777+0000","last_active":"2024-04-25T11:45:16.734685+0000","last_peered":"2024-04-25T11:45:16.734685+0000","last_clean":"2024-04-25T11:45:16.734685+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:45:16.734685+0000","last_undegraded":"2024-04-25T11:45:16.734685+0000","last_fullsized":"2024-04-25T11:45:16.734685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'1093","last_scrub_stamp":"2024-04-25T11:45:16.734476+0000","last_deep_scrub":"40'1093","last_deep_scrub_stamp":"2024-04-25T11:45:16.734476+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.734476+0000","objects_scrubbed":3,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:03.233019+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5159","reported_seq":5267,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.480064+0000","last_change":"2024-04-25T11:45:16.480096+0000","last_active":"2024-04-25T11:45:16.480064+0000","last_peered":"2024-04-25T11:45:16.480064+0000","last_clean":"2024-04-25T11:45:16.480064+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:16.480064+0000","last_undegraded":"2024-04-25T11:45:16.480064+0000","last_fullsized":"2024-04-25T11:45:16.480064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5159","last_scrub_stamp":"2024-04-25T11:45:16.479962+0000","last_deep_scrub":"40'5159","last_deep_scrub_stamp":"2024-04-25T11:45:16.479962+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.479962+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:38.490567+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21468,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"40'9","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.837718+0000","last_change":"2024-04-25T11:45:19.837792+0000","last_active":"2024-04-25T11:45:19.837718+0000","last_peered":"2024-04-25T11:45:19.837718+0000","last_clean":"2024-04-25T11:45:19.837718+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:45:19.837718+0000","last_undegraded":"2024-04-25T11:45:19.837718+0000","last_fullsized":"2024-04-25T11:45:19.837718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:19.837486+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:19.837486+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.837486+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:07.542967+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"40'15","reported_seq":94,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.042850+0000","last_change":"2024-04-25T11:45:09.042888+0000","last_active":"2024-04-25T11:45:09.042850+0000","last_peered":"2024-04-25T11:45:09.042850+0000","last_clean":"2024-04-25T11:45:09.042850+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:45:09.042850+0000","last_undegraded":"2024-04-25T11:45:09.042850+0000","last_fullsized":"2024-04-25T11:45:09.042850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.042755+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.042755+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.042755+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:28.867840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5171","reported_seq":5278,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.399544+0000","last_change":"2024-04-25T11:45:22.399653+0000","last_active":"2024-04-25T11:45:22.399544+0000","last_peered":"2024-04-25T11:45:22.399544+0000","last_clean":"2024-04-25T11:45:22.399544+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:22.399544+0000","last_undegraded":"2024-04-25T11:45:22.399544+0000","last_fullsized":"2024-04-25T11:45:22.399544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5171","last_scrub_stamp":"2024-04-25T11:45:22.399259+0000","last_deep_scrub":"40'5171","last_deep_scrub_stamp":"2024-04-25T11:45:22.399259+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.399259+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.919447+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21566,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.747264+0000","last_change":"2024-04-25T11:45:18.747314+0000","last_active":"2024-04-25T11:45:18.747264+0000","last_peered":"2024-04-25T11:45:18.747264+0000","last_clean":"2024-04-25T11:45:18.747264+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:45:18.747264+0000","last_undegraded":"2024-04-25T11:45:18.747264+0000","last_fullsized":"2024-04-25T11:45:18.747264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:18.747128+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:18.747128+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.747128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:35.565657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"40'13","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.953773+0000","last_change":"2024-04-25T11:45:10.953852+0000","last_active":"2024-04-25T11:45:10.953773+0000","last_peered":"2024-04-25T11:45:10.953773+0000","last_clean":"2024-04-25T11:45:10.953773+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:45:10.953773+0000","last_undegraded":"2024-04-25T11:45:10.953773+0000","last_fullsized":"2024-04-25T11:45:10.953773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:10.953598+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:10.953598+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.953598+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.613663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6717,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.840640+0000","last_change":"2024-04-25T11:45:16.840671+0000","last_active":"2024-04-25T11:45:16.840640+0000","last_peered":"2024-04-25T11:45:16.840640+0000","last_clean":"2024-04-25T11:45:16.840640+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:16.840640+0000","last_undegraded":"2024-04-25T11:45:16.840640+0000","last_fullsized":"2024-04-25T11:45:16.840640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6596","last_scrub_stamp":"2024-04-25T11:45:16.840533+0000","last_deep_scrub":"40'6596","last_deep_scrub_stamp":"2024-04-25T11:45:16.840533+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.840533+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:18.318443+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.333845+0000","last_change":"2024-04-25T11:45:10.333911+0000","last_active":"2024-04-25T11:45:10.333845+0000","last_peered":"2024-04-25T11:45:10.333845+0000","last_clean":"2024-04-25T11:45:10.333845+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:45:10.333845+0000","last_undegraded":"2024-04-25T11:45:10.333845+0000","last_fullsized":"2024-04-25T11:45:10.333845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:10.333643+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:10.333643+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.333643+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:52.557097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'8","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.646042+0000","last_change":"2024-04-25T11:45:12.646193+0000","last_active":"2024-04-25T11:45:12.646042+0000","last_peered":"2024-04-25T11:45:12.646042+0000","last_clean":"2024-04-25T11:45:12.646042+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:45:12.646042+0000","last_undegraded":"2024-04-25T11:45:12.646042+0000","last_fullsized":"2024-04-25T11:45:12.646042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:12.645732+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:12.645732+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.645732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:28.657080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4795","reported_seq":4901,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.515960+0000","last_change":"2024-04-25T11:45:17.515990+0000","last_active":"2024-04-25T11:45:17.515960+0000","last_peered":"2024-04-25T11:45:17.515960+0000","last_clean":"2024-04-25T11:45:17.515960+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:17.515960+0000","last_undegraded":"2024-04-25T11:45:17.515960+0000","last_fullsized":"2024-04-25T11:45:17.515960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4795","last_scrub_stamp":"2024-04-25T11:45:17.515869+0000","last_deep_scrub":"40'4795","last_deep_scrub_stamp":"2024-04-25T11:45:17.515869+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.515869+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:40.213666+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19675,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'18","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:06.965535+0000","last_change":"2024-04-25T11:45:06.965581+0000","last_active":"2024-04-25T11:45:06.965535+0000","last_peered":"2024-04-25T11:45:06.965535+0000","last_clean":"2024-04-25T11:45:06.965535+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:45:06.965535+0000","last_undegraded":"2024-04-25T11:45:06.965535+0000","last_fullsized":"2024-04-25T11:45:06.965535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:06.965415+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:06.965415+0000","last_clean_scrub_stamp":"2024-04-25T11:45:06.965415+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:27:21.084447+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'18","reported_seq":114,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.407459+0000","last_change":"2024-04-25T11:45:13.407575+0000","last_active":"2024-04-25T11:45:13.407459+0000","last_peered":"2024-04-25T11:45:13.407459+0000","last_clean":"2024-04-25T11:45:13.407459+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:45:13.407459+0000","last_undegraded":"2024-04-25T11:45:13.407459+0000","last_fullsized":"2024-04-25T11:45:13.407459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:13.407199+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:13.407199+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.407199+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:12.797988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4482","reported_seq":4593,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.385961+0000","last_change":"2024-04-25T11:45:23.385961+0000","last_active":"2024-04-25T11:45:23.385961+0000","last_peered":"2024-04-25T11:45:23.385961+0000","last_clean":"2024-04-25T11:45:23.385961+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:23.385961+0000","last_undegraded":"2024-04-25T11:45:23.385961+0000","last_fullsized":"2024-04-25T11:45:23.385961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4482","last_scrub_stamp":"2024-04-25T11:45:23.385828+0000","last_deep_scrub":"40'4482","last_deep_scrub_stamp":"2024-04-25T11:45:23.385828+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.385828+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:34.129470+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18513,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.880339+0000","last_change":"2024-04-25T11:45:20.880417+0000","last_active":"2024-04-25T11:45:20.880339+0000","last_peered":"2024-04-25T11:45:20.880339+0000","last_clean":"2024-04-25T11:45:20.880339+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:45:20.880339+0000","last_undegraded":"2024-04-25T11:45:20.880339+0000","last_fullsized":"2024-04-25T11:45:20.880339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:20.880111+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:20.880111+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.880111+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:08.095052+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'372","reported_seq":461,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.936453+0000","last_change":"2024-04-25T11:45:11.936609+0000","last_active":"2024-04-25T11:45:11.936453+0000","last_peered":"2024-04-25T11:45:11.936453+0000","last_clean":"2024-04-25T11:45:11.936453+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:45:11.936453+0000","last_undegraded":"2024-04-25T11:45:11.936453+0000","last_fullsized":"2024-04-25T11:45:11.936453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'372","last_scrub_stamp":"2024-04-25T11:45:11.936219+0000","last_deep_scrub":"40'372","last_deep_scrub_stamp":"2024-04-25T11:45:11.936219+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.936219+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:11.984038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4559","reported_seq":4668,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:26.915942+0000","last_change":"2024-04-25T11:45:26.916023+0000","last_active":"2024-04-25T11:45:26.915942+0000","last_peered":"2024-04-25T11:45:26.915942+0000","last_clean":"2024-04-25T11:45:26.915942+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:45:26.915942+0000","last_undegraded":"2024-04-25T11:45:26.915942+0000","last_fullsized":"2024-04-25T11:45:26.915942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4559","last_scrub_stamp":"2024-04-25T11:45:26.915685+0000","last_deep_scrub":"40'4559","last_deep_scrub_stamp":"2024-04-25T11:45:26.915685+0000","last_clean_scrub_stamp":"2024-04-25T11:45:26.915685+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:55.236848+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19035,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'9","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.906484+0000","last_change":"2024-04-25T11:45:15.906560+0000","last_active":"2024-04-25T11:45:15.906484+0000","last_peered":"2024-04-25T11:45:15.906484+0000","last_clean":"2024-04-25T11:45:15.906484+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:45:15.906484+0000","last_undegraded":"2024-04-25T11:45:15.906484+0000","last_fullsized":"2024-04-25T11:45:15.906484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:15.906213+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:15.906213+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.906213+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:02.404755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'9","reported_seq":95,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.709994+0000","last_change":"2024-04-25T11:45:14.709994+0000","last_active":"2024-04-25T11:45:14.709994+0000","last_peered":"2024-04-25T11:45:14.709994+0000","last_clean":"2024-04-25T11:45:14.709994+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:45:14.709994+0000","last_undegraded":"2024-04-25T11:45:14.709994+0000","last_fullsized":"2024-04-25T11:45:14.709994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:14.709707+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:14.709707+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.709707+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:36.012017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5085","reported_seq":5195,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.227257+0000","last_change":"2024-04-25T11:45:21.227318+0000","last_active":"2024-04-25T11:45:21.227257+0000","last_peered":"2024-04-25T11:45:21.227257+0000","last_clean":"2024-04-25T11:45:21.227257+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:45:21.227257+0000","last_undegraded":"2024-04-25T11:45:21.227257+0000","last_fullsized":"2024-04-25T11:45:21.227257+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5085","last_scrub_stamp":"2024-04-25T11:45:21.227103+0000","last_deep_scrub":"40'5085","last_deep_scrub_stamp":"2024-04-25T11:45:21.227103+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.227103+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:58.346166+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21093,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'14","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.186717+0000","last_change":"2024-04-25T11:45:22.186717+0000","last_active":"2024-04-25T11:45:22.186717+0000","last_peered":"2024-04-25T11:45:22.186717+0000","last_clean":"2024-04-25T11:45:22.186717+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:45:22.186717+0000","last_undegraded":"2024-04-25T11:45:22.186717+0000","last_fullsized":"2024-04-25T11:45:22.186717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:22.186318+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:22.186318+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.186318+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:09.891799+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"40'5","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.037300+0000","last_change":"2024-04-25T11:45:08.037323+0000","last_active":"2024-04-25T11:45:08.037300+0000","last_peered":"2024-04-25T11:45:08.037300+0000","last_clean":"2024-04-25T11:45:08.037300+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:45:08.037300+0000","last_undegraded":"2024-04-25T11:45:08.037300+0000","last_fullsized":"2024-04-25T11:45:08.037300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:08.037222+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:08.037222+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.037222+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:41.353562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5304","reported_seq":5420,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.914766+0000","last_change":"2024-04-25T11:45:20.914766+0000","last_active":"2024-04-25T11:45:20.914766+0000","last_peered":"2024-04-25T11:45:20.914766+0000","last_clean":"2024-04-25T11:45:20.914766+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:20.914766+0000","last_undegraded":"2024-04-25T11:45:20.914766+0000","last_fullsized":"2024-04-25T11:45:20.914766+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5304","last_scrub_stamp":"2024-04-25T11:45:20.914252+0000","last_deep_scrub":"40'5304","last_deep_scrub_stamp":"2024-04-25T11:45:20.914252+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.914252+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:58.033315+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21947,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.289546+0000","last_change":"2024-04-25T11:45:12.289546+0000","last_active":"2024-04-25T11:45:12.289546+0000","last_peered":"2024-04-25T11:45:12.289546+0000","last_clean":"2024-04-25T11:45:12.289546+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:45:12.289546+0000","last_undegraded":"2024-04-25T11:45:12.289546+0000","last_fullsized":"2024-04-25T11:45:12.289546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:12.289275+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:12.289275+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.289275+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:59.622600+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'17","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.212252+0000","last_change":"2024-04-25T11:45:11.212252+0000","last_active":"2024-04-25T11:45:11.212252+0000","last_peered":"2024-04-25T11:45:11.212252+0000","last_clean":"2024-04-25T11:45:11.212252+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:45:11.212252+0000","last_undegraded":"2024-04-25T11:45:11.212252+0000","last_fullsized":"2024-04-25T11:45:11.212252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.211966+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.211966+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.211966+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:33.726984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5442","reported_seq":5561,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.959541+0000","last_change":"2024-04-25T11:45:19.959686+0000","last_active":"2024-04-25T11:45:19.959541+0000","last_peered":"2024-04-25T11:45:19.959541+0000","last_clean":"2024-04-25T11:45:19.959541+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:19.959541+0000","last_undegraded":"2024-04-25T11:45:19.959541+0000","last_fullsized":"2024-04-25T11:45:19.959541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5442","last_scrub_stamp":"2024-04-25T11:45:19.959160+0000","last_deep_scrub":"40'5442","last_deep_scrub_stamp":"2024-04-25T11:45:19.959160+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.959160+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:30.702802+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23011,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"40'12","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.903319+0000","last_change":"2024-04-25T11:45:13.903450+0000","last_active":"2024-04-25T11:45:13.903319+0000","last_peered":"2024-04-25T11:45:13.903319+0000","last_clean":"2024-04-25T11:45:13.903319+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:45:13.903319+0000","last_undegraded":"2024-04-25T11:45:13.903319+0000","last_fullsized":"2024-04-25T11:45:13.903319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:13.903084+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:13.903084+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.903084+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:35.913689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'66","reported_seq":201,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.590324+0000","last_change":"2024-04-25T11:45:07.590324+0000","last_active":"2024-04-25T11:45:07.590324+0000","last_peered":"2024-04-25T11:45:07.590324+0000","last_clean":"2024-04-25T11:45:07.590324+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:07.590324+0000","last_undegraded":"2024-04-25T11:45:07.590324+0000","last_fullsized":"2024-04-25T11:45:07.590324+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'66","last_scrub_stamp":"2024-04-25T11:45:07.590202+0000","last_deep_scrub":"40'66","last_deep_scrub_stamp":"2024-04-25T11:45:07.590202+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.590202+0000","objects_scrubbed":4,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:35.941465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5066","reported_seq":5181,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.056629+0000","last_change":"2024-04-25T11:45:18.056774+0000","last_active":"2024-04-25T11:45:18.056629+0000","last_peered":"2024-04-25T11:45:18.056629+0000","last_clean":"2024-04-25T11:45:18.056629+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:18.056629+0000","last_undegraded":"2024-04-25T11:45:18.056629+0000","last_fullsized":"2024-04-25T11:45:18.056629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5066","last_scrub_stamp":"2024-04-25T11:45:18.056251+0000","last_deep_scrub":"40'5066","last_deep_scrub_stamp":"2024-04-25T11:45:18.056251+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.056251+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:01.524201+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21672,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'6","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.998276+0000","last_change":"2024-04-25T11:45:13.998441+0000","last_active":"2024-04-25T11:45:13.998276+0000","last_peered":"2024-04-25T11:45:13.998276+0000","last_clean":"2024-04-25T11:45:13.998276+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:45:13.998276+0000","last_undegraded":"2024-04-25T11:45:13.998276+0000","last_fullsized":"2024-04-25T11:45:13.998276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:13.998024+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:13.998024+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.998024+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:36.008629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'19","reported_seq":98,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.667771+0000","last_change":"2024-04-25T11:45:13.667844+0000","last_active":"2024-04-25T11:45:13.667771+0000","last_peered":"2024-04-25T11:45:13.667771+0000","last_clean":"2024-04-25T11:45:13.667771+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:45:13.667771+0000","last_undegraded":"2024-04-25T11:45:13.667771+0000","last_fullsized":"2024-04-25T11:45:13.667771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'19","last_scrub_stamp":"2024-04-25T11:45:13.667567+0000","last_deep_scrub":"40'19","last_deep_scrub_stamp":"2024-04-25T11:45:13.667567+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.667567+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:49:10.180138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5144","reported_seq":5249,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.261794+0000","last_change":"2024-04-25T11:45:20.261944+0000","last_active":"2024-04-25T11:45:20.261794+0000","last_peered":"2024-04-25T11:45:20.261794+0000","last_clean":"2024-04-25T11:45:20.261794+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:20.261794+0000","last_undegraded":"2024-04-25T11:45:20.261794+0000","last_fullsized":"2024-04-25T11:45:20.261794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5144","last_scrub_stamp":"2024-04-25T11:45:20.261504+0000","last_deep_scrub":"40'5144","last_deep_scrub_stamp":"2024-04-25T11:45:20.261504+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.261504+0000","objects_scrubbed":0,"log_size":5144,"log_dups_size":0,"ondisk_log_size":5144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:31.005146+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21523,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"40'5","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.715935+0000","last_change":"2024-04-25T11:45:11.715935+0000","last_active":"2024-04-25T11:45:11.715935+0000","last_peered":"2024-04-25T11:45:11.715935+0000","last_clean":"2024-04-25T11:45:11.715935+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:45:11.715935+0000","last_undegraded":"2024-04-25T11:45:11.715935+0000","last_fullsized":"2024-04-25T11:45:11.715935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:11.715767+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:11.715767+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.715767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:08.950473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"40'14","reported_seq":105,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.225628+0000","last_change":"2024-04-25T11:45:10.225628+0000","last_active":"2024-04-25T11:45:10.225628+0000","last_peered":"2024-04-25T11:45:10.225628+0000","last_clean":"2024-04-25T11:45:10.225628+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:45:10.225628+0000","last_undegraded":"2024-04-25T11:45:10.225628+0000","last_fullsized":"2024-04-25T11:45:10.225628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:10.225203+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:10.225203+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.225203+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:10.273022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"40'74","reported_seq":173,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.679843+0000","last_change":"2024-04-25T11:45:11.679928+0000","last_active":"2024-04-25T11:45:11.679843+0000","last_peered":"2024-04-25T11:45:11.679843+0000","last_clean":"2024-04-25T11:45:11.679843+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:45:11.679843+0000","last_undegraded":"2024-04-25T11:45:11.679843+0000","last_fullsized":"2024-04-25T11:45:11.679843+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'74","last_scrub_stamp":"2024-04-25T11:45:11.679639+0000","last_deep_scrub":"40'74","last_deep_scrub_stamp":"2024-04-25T11:45:11.679639+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.679639+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:19.095500+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5040","reported_seq":5144,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.677896+0000","last_change":"2024-04-25T11:45:24.678036+0000","last_active":"2024-04-25T11:45:24.677896+0000","last_peered":"2024-04-25T11:45:24.677896+0000","last_clean":"2024-04-25T11:45:24.677896+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:24.677896+0000","last_undegraded":"2024-04-25T11:45:24.677896+0000","last_fullsized":"2024-04-25T11:45:24.677896+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5040","last_scrub_stamp":"2024-04-25T11:45:24.677516+0000","last_deep_scrub":"40'5040","last_deep_scrub_stamp":"2024-04-25T11:45:24.677516+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.677516+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:35.421158+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21819,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'14","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.739326+0000","last_change":"2024-04-25T11:45:22.739455+0000","last_active":"2024-04-25T11:45:22.739326+0000","last_peered":"2024-04-25T11:45:22.739326+0000","last_clean":"2024-04-25T11:45:22.739326+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:45:22.739326+0000","last_undegraded":"2024-04-25T11:45:22.739326+0000","last_fullsized":"2024-04-25T11:45:22.739326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:22.739061+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:22.739061+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.739061+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:34.960612+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'18","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.634632+0000","last_change":"2024-04-25T11:45:10.634731+0000","last_active":"2024-04-25T11:45:10.634632+0000","last_peered":"2024-04-25T11:45:10.634632+0000","last_clean":"2024-04-25T11:45:10.634632+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:10.634632+0000","last_undegraded":"2024-04-25T11:45:10.634632+0000","last_fullsized":"2024-04-25T11:45:10.634632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:10.634413+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:10.634413+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.634413+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:34.877637+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5278","reported_seq":5385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.724072+0000","last_change":"2024-04-25T11:45:23.724119+0000","last_active":"2024-04-25T11:45:23.724072+0000","last_peered":"2024-04-25T11:45:23.724072+0000","last_clean":"2024-04-25T11:45:23.724072+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:23.724072+0000","last_undegraded":"2024-04-25T11:45:23.724072+0000","last_fullsized":"2024-04-25T11:45:23.724072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5278","last_scrub_stamp":"2024-04-25T11:45:23.723931+0000","last_deep_scrub":"40'5278","last_deep_scrub_stamp":"2024-04-25T11:45:23.723931+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.723931+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.244119+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22099,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.093007+0000","last_change":"2024-04-25T11:45:19.093141+0000","last_active":"2024-04-25T11:45:19.093007+0000","last_peered":"2024-04-25T11:45:19.093007+0000","last_clean":"2024-04-25T11:45:19.093007+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:45:19.093007+0000","last_undegraded":"2024-04-25T11:45:19.093007+0000","last_fullsized":"2024-04-25T11:45:19.093007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:19.092669+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:19.092669+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.092669+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:36.520409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'15","reported_seq":92,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.997475+0000","last_change":"2024-04-25T11:45:09.997607+0000","last_active":"2024-04-25T11:45:09.997475+0000","last_peered":"2024-04-25T11:45:09.997475+0000","last_clean":"2024-04-25T11:45:09.997475+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:45:09.997475+0000","last_undegraded":"2024-04-25T11:45:09.997475+0000","last_fullsized":"2024-04-25T11:45:09.997475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.997165+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.997165+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.997165+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:40.884137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4958","reported_seq":5067,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.185146+0000","last_change":"2024-04-25T11:45:23.185177+0000","last_active":"2024-04-25T11:45:23.185146+0000","last_peered":"2024-04-25T11:45:23.185146+0000","last_clean":"2024-04-25T11:45:23.185146+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:23.185146+0000","last_undegraded":"2024-04-25T11:45:23.185146+0000","last_fullsized":"2024-04-25T11:45:23.185146+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4958","last_scrub_stamp":"2024-04-25T11:45:23.185041+0000","last_deep_scrub":"40'4958","last_deep_scrub_stamp":"2024-04-25T11:45:23.185041+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.185041+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:10.890522+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20958,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.870627+0000","last_change":"2024-04-25T11:45:25.870658+0000","last_active":"2024-04-25T11:45:25.870627+0000","last_peered":"2024-04-25T11:45:25.870627+0000","last_clean":"2024-04-25T11:45:25.870627+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:45:25.870627+0000","last_undegraded":"2024-04-25T11:45:25.870627+0000","last_fullsized":"2024-04-25T11:45:25.870627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:25.870527+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:25.870527+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.870527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:20.991868+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'14","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.662930+0000","last_change":"2024-04-25T11:45:09.662975+0000","last_active":"2024-04-25T11:45:09.662930+0000","last_peered":"2024-04-25T11:45:09.662930+0000","last_clean":"2024-04-25T11:45:09.662930+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:09.662930+0000","last_undegraded":"2024-04-25T11:45:09.662930+0000","last_fullsized":"2024-04-25T11:45:09.662930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:09.662819+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:09.662819+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.662819+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:09.907533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4822","reported_seq":4934,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.354677+0000","last_change":"2024-04-25T11:45:25.354732+0000","last_active":"2024-04-25T11:45:25.354677+0000","last_peered":"2024-04-25T11:45:25.354677+0000","last_clean":"2024-04-25T11:45:25.354677+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:25.354677+0000","last_undegraded":"2024-04-25T11:45:25.354677+0000","last_fullsized":"2024-04-25T11:45:25.354677+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4822","last_scrub_stamp":"2024-04-25T11:45:25.354535+0000","last_deep_scrub":"40'4822","last_deep_scrub_stamp":"2024-04-25T11:45:25.354535+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.354535+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:45:57.712074+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20365,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'8","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.749429+0000","last_change":"2024-04-25T11:45:23.749511+0000","last_active":"2024-04-25T11:45:23.749429+0000","last_peered":"2024-04-25T11:45:23.749429+0000","last_clean":"2024-04-25T11:45:23.749429+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:45:23.749429+0000","last_undegraded":"2024-04-25T11:45:23.749429+0000","last_fullsized":"2024-04-25T11:45:23.749429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:23.749274+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:23.749274+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.749274+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:23.304151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'8","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.304458+0000","last_change":"2024-04-25T11:45:09.304533+0000","last_active":"2024-04-25T11:45:09.304458+0000","last_peered":"2024-04-25T11:45:09.304458+0000","last_clean":"2024-04-25T11:45:09.304458+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:45:09.304458+0000","last_undegraded":"2024-04-25T11:45:09.304458+0000","last_fullsized":"2024-04-25T11:45:09.304458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:09.304266+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:09.304266+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.304266+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:34.893006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4349","reported_seq":4453,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.384996+0000","last_change":"2024-04-25T11:45:21.385042+0000","last_active":"2024-04-25T11:45:21.384996+0000","last_peered":"2024-04-25T11:45:21.384996+0000","last_clean":"2024-04-25T11:45:21.384996+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:21.384996+0000","last_undegraded":"2024-04-25T11:45:21.384996+0000","last_fullsized":"2024-04-25T11:45:21.384996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4349","last_scrub_stamp":"2024-04-25T11:45:21.384877+0000","last_deep_scrub":"40'4349","last_deep_scrub_stamp":"2024-04-25T11:45:21.384877+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.384877+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:04.852828+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18465,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.719257+0000","last_change":"2024-04-25T11:45:20.719329+0000","last_active":"2024-04-25T11:45:20.719257+0000","last_peered":"2024-04-25T11:45:20.719257+0000","last_clean":"2024-04-25T11:45:20.719257+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:45:20.719257+0000","last_undegraded":"2024-04-25T11:45:20.719257+0000","last_fullsized":"2024-04-25T11:45:20.719257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:20.719027+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:20.719027+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.719027+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:24.521751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'39","reported_seq":143,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.635646+0000","last_change":"2024-04-25T11:45:08.635716+0000","last_active":"2024-04-25T11:45:08.635646+0000","last_peered":"2024-04-25T11:45:08.635646+0000","last_clean":"2024-04-25T11:45:08.635646+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:08.635646+0000","last_undegraded":"2024-04-25T11:45:08.635646+0000","last_fullsized":"2024-04-25T11:45:08.635646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'39","last_scrub_stamp":"2024-04-25T11:45:08.635551+0000","last_deep_scrub":"40'39","last_deep_scrub_stamp":"2024-04-25T11:45:08.635551+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.635551+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:14.634807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5276","reported_seq":5383,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.278283+0000","last_change":"2024-04-25T11:45:19.278363+0000","last_active":"2024-04-25T11:45:19.278283+0000","last_peered":"2024-04-25T11:45:19.278283+0000","last_clean":"2024-04-25T11:45:19.278283+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:45:19.278283+0000","last_undegraded":"2024-04-25T11:45:19.278283+0000","last_fullsized":"2024-04-25T11:45:19.278283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5276","last_scrub_stamp":"2024-04-25T11:45:19.278149+0000","last_deep_scrub":"40'5276","last_deep_scrub_stamp":"2024-04-25T11:45:19.278149+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.278149+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:47.798338+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21946,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.457209+0000","last_change":"2024-04-25T11:45:15.457365+0000","last_active":"2024-04-25T11:45:15.457209+0000","last_peered":"2024-04-25T11:45:15.457209+0000","last_clean":"2024-04-25T11:45:15.457209+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:45:15.457209+0000","last_undegraded":"2024-04-25T11:45:15.457209+0000","last_fullsized":"2024-04-25T11:45:15.457209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:15.456978+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:15.456978+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.456978+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:22.872839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'18","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.403829+0000","last_change":"2024-04-25T11:45:11.403927+0000","last_active":"2024-04-25T11:45:11.403829+0000","last_peered":"2024-04-25T11:45:11.403829+0000","last_clean":"2024-04-25T11:45:11.403829+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:45:11.403829+0000","last_undegraded":"2024-04-25T11:45:11.403829+0000","last_fullsized":"2024-04-25T11:45:11.403829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:11.403629+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:11.403629+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.403629+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:53.627083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5730","reported_seq":5856,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.946836+0000","last_change":"2024-04-25T11:45:23.946901+0000","last_active":"2024-04-25T11:45:23.946836+0000","last_peered":"2024-04-25T11:45:23.946836+0000","last_clean":"2024-04-25T11:45:23.946836+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:45:23.946836+0000","last_undegraded":"2024-04-25T11:45:23.946836+0000","last_fullsized":"2024-04-25T11:45:23.946836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5730","last_scrub_stamp":"2024-04-25T11:45:23.946688+0000","last_deep_scrub":"40'5730","last_deep_scrub_stamp":"2024-04-25T11:45:23.946688+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.946688+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:52:02.885338+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23847,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.741050+0000","last_change":"2024-04-25T11:45:24.741121+0000","last_active":"2024-04-25T11:45:24.741050+0000","last_peered":"2024-04-25T11:45:24.741050+0000","last_clean":"2024-04-25T11:45:24.741050+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:45:24.741050+0000","last_undegraded":"2024-04-25T11:45:24.741050+0000","last_fullsized":"2024-04-25T11:45:24.741050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:24.740857+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:24.740857+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.740857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:00.538047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"40'14","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.173175+0000","last_change":"2024-04-25T11:45:13.173269+0000","last_active":"2024-04-25T11:45:13.173175+0000","last_peered":"2024-04-25T11:45:13.173175+0000","last_clean":"2024-04-25T11:45:13.173175+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:45:13.173175+0000","last_undegraded":"2024-04-25T11:45:13.173175+0000","last_fullsized":"2024-04-25T11:45:13.173175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:13.172976+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:13.172976+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.172976+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:35.428316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4272","reported_seq":4389,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.906529+0000","last_change":"2024-04-25T11:45:24.906659+0000","last_active":"2024-04-25T11:45:24.906529+0000","last_peered":"2024-04-25T11:45:24.906529+0000","last_clean":"2024-04-25T11:45:24.906529+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:45:24.906529+0000","last_undegraded":"2024-04-25T11:45:24.906529+0000","last_fullsized":"2024-04-25T11:45:24.906529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4272","last_scrub_stamp":"2024-04-25T11:45:24.906194+0000","last_deep_scrub":"40'4272","last_deep_scrub_stamp":"2024-04-25T11:45:24.906194+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.906194+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:12.398388+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17563,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.994561+0000","last_change":"2024-04-25T11:45:11.994699+0000","last_active":"2024-04-25T11:45:11.994561+0000","last_peered":"2024-04-25T11:45:11.994561+0000","last_clean":"2024-04-25T11:45:11.994561+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:45:11.994561+0000","last_undegraded":"2024-04-25T11:45:11.994561+0000","last_fullsized":"2024-04-25T11:45:11.994561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:11.994331+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:11.994331+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.994331+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:12.042150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'19","reported_seq":117,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.932211+0000","last_change":"2024-04-25T11:45:07.932248+0000","last_active":"2024-04-25T11:45:07.932211+0000","last_peered":"2024-04-25T11:45:07.932211+0000","last_clean":"2024-04-25T11:45:07.932211+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:45:07.932211+0000","last_undegraded":"2024-04-25T11:45:07.932211+0000","last_fullsized":"2024-04-25T11:45:07.932211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'19","last_scrub_stamp":"2024-04-25T11:45:07.932138+0000","last_deep_scrub":"40'19","last_deep_scrub_stamp":"2024-04-25T11:45:07.932138+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.932138+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:41.248478+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4282","reported_seq":4385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.369522+0000","last_change":"2024-04-25T11:45:24.369563+0000","last_active":"2024-04-25T11:45:24.369522+0000","last_peered":"2024-04-25T11:45:24.369522+0000","last_clean":"2024-04-25T11:45:24.369522+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:24.369522+0000","last_undegraded":"2024-04-25T11:45:24.369522+0000","last_fullsized":"2024-04-25T11:45:24.369522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4282","last_scrub_stamp":"2024-04-25T11:45:24.369372+0000","last_deep_scrub":"40'4282","last_deep_scrub_stamp":"2024-04-25T11:45:24.369372+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.369372+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:27.595641+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17949,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.711671+0000","last_change":"2024-04-25T11:45:21.711805+0000","last_active":"2024-04-25T11:45:21.711671+0000","last_peered":"2024-04-25T11:45:21.711671+0000","last_clean":"2024-04-25T11:45:21.711671+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:45:21.711671+0000","last_undegraded":"2024-04-25T11:45:21.711671+0000","last_fullsized":"2024-04-25T11:45:21.711671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.711369+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.711369+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.711369+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:39.139108+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'353","reported_seq":455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.456243+0000","last_change":"2024-04-25T11:45:10.456343+0000","last_active":"2024-04-25T11:45:10.456243+0000","last_peered":"2024-04-25T11:45:10.456243+0000","last_clean":"2024-04-25T11:45:10.456243+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:45:10.456243+0000","last_undegraded":"2024-04-25T11:45:10.456243+0000","last_fullsized":"2024-04-25T11:45:10.456243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'353","last_scrub_stamp":"2024-04-25T11:45:10.456029+0000","last_deep_scrub":"40'353","last_deep_scrub_stamp":"2024-04-25T11:45:10.456029+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.456029+0000","objects_scrubbed":6,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:50.007081+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4220","reported_seq":4327,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.984701+0000","last_change":"2024-04-25T11:45:18.984744+0000","last_active":"2024-04-25T11:45:18.984701+0000","last_peered":"2024-04-25T11:45:18.984701+0000","last_clean":"2024-04-25T11:45:18.984701+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:18.984701+0000","last_undegraded":"2024-04-25T11:45:18.984701+0000","last_fullsized":"2024-04-25T11:45:18.984701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4220","last_scrub_stamp":"2024-04-25T11:45:18.984548+0000","last_deep_scrub":"40'4220","last_deep_scrub_stamp":"2024-04-25T11:45:18.984548+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.984548+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:47.504736+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17734,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.248931+0000","last_change":"2024-04-25T11:45:13.249000+0000","last_active":"2024-04-25T11:45:13.248931+0000","last_peered":"2024-04-25T11:45:13.248931+0000","last_clean":"2024-04-25T11:45:13.248931+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:45:13.248931+0000","last_undegraded":"2024-04-25T11:45:13.248931+0000","last_fullsized":"2024-04-25T11:45:13.248931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.248736+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.248736+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.248736+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:52.101758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'33","reported_seq":117,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.932343+0000","last_change":"2024-04-25T11:45:12.932422+0000","last_active":"2024-04-25T11:45:12.932343+0000","last_peered":"2024-04-25T11:45:12.932343+0000","last_clean":"2024-04-25T11:45:12.932343+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:45:12.932343+0000","last_undegraded":"2024-04-25T11:45:12.932343+0000","last_fullsized":"2024-04-25T11:45:12.932343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'33","last_scrub_stamp":"2024-04-25T11:45:12.932164+0000","last_deep_scrub":"40'33","last_deep_scrub_stamp":"2024-04-25T11:45:12.932164+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.932164+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:35.447183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":32,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4473","reported_seq":4571,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.444385+0000","last_change":"2024-04-25T11:45:18.444414+0000","last_active":"2024-04-25T11:45:18.444385+0000","last_peered":"2024-04-25T11:45:18.444385+0000","last_clean":"2024-04-25T11:45:18.444385+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:18.444385+0000","last_undegraded":"2024-04-25T11:45:18.444385+0000","last_fullsized":"2024-04-25T11:45:18.444385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4473","last_scrub_stamp":"2024-04-25T11:45:18.444279+0000","last_deep_scrub":"40'4473","last_deep_scrub_stamp":"2024-04-25T11:45:18.444279+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.444279+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:15.678984+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18511,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.155314+0000","last_change":"2024-04-25T11:45:16.155376+0000","last_active":"2024-04-25T11:45:16.155314+0000","last_peered":"2024-04-25T11:45:16.155314+0000","last_clean":"2024-04-25T11:45:16.155314+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:45:16.155314+0000","last_undegraded":"2024-04-25T11:45:16.155314+0000","last_fullsized":"2024-04-25T11:45:16.155314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:16.155116+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:16.155116+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.155116+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:02.653658+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"40'9","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.175940+0000","last_change":"2024-04-25T11:45:14.176029+0000","last_active":"2024-04-25T11:45:14.175940+0000","last_peered":"2024-04-25T11:45:14.175940+0000","last_clean":"2024-04-25T11:45:14.175940+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:45:14.175940+0000","last_undegraded":"2024-04-25T11:45:14.175940+0000","last_fullsized":"2024-04-25T11:45:14.175940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:14.175731+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:14.175731+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.175731+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:11.410437+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4342","reported_seq":4454,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:28.839558+0000","last_change":"2024-04-25T11:45:28.839608+0000","last_active":"2024-04-25T11:45:28.839558+0000","last_peered":"2024-04-25T11:45:28.839558+0000","last_clean":"2024-04-25T11:45:28.839558+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:45:28.839558+0000","last_undegraded":"2024-04-25T11:45:28.839558+0000","last_fullsized":"2024-04-25T11:45:28.839558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4342","last_scrub_stamp":"2024-04-25T11:45:28.839445+0000","last_deep_scrub":"40'4342","last_deep_scrub_stamp":"2024-04-25T11:45:28.839445+0000","last_clean_scrub_stamp":"2024-04-25T11:45:28.839445+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:56.510470+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17952,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.187114+0000","last_change":"2024-04-25T11:45:15.187264+0000","last_active":"2024-04-25T11:45:15.187114+0000","last_peered":"2024-04-25T11:45:15.187114+0000","last_clean":"2024-04-25T11:45:15.187114+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:45:15.187114+0000","last_undegraded":"2024-04-25T11:45:15.187114+0000","last_fullsized":"2024-04-25T11:45:15.187114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:15.186774+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:15.186774+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.186774+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:24.739444+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'17","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.439896+0000","last_change":"2024-04-25T11:45:14.439974+0000","last_active":"2024-04-25T11:45:14.439896+0000","last_peered":"2024-04-25T11:45:14.439896+0000","last_clean":"2024-04-25T11:45:14.439896+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:45:14.439896+0000","last_undegraded":"2024-04-25T11:45:14.439896+0000","last_fullsized":"2024-04-25T11:45:14.439896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:14.439671+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:14.439671+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.439671+0000","objects_scrubbed":3,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:53.292692+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3797","reported_seq":3908,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.852413+0000","last_change":"2024-04-25T11:45:25.852453+0000","last_active":"2024-04-25T11:45:25.852413+0000","last_peered":"2024-04-25T11:45:25.852413+0000","last_clean":"2024-04-25T11:45:25.852413+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:45:25.852413+0000","last_undegraded":"2024-04-25T11:45:25.852413+0000","last_fullsized":"2024-04-25T11:45:25.852413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'3797","last_scrub_stamp":"2024-04-25T11:45:25.852274+0000","last_deep_scrub":"40'3797","last_deep_scrub_stamp":"2024-04-25T11:45:25.852274+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.852274+0000","objects_scrubbed":0,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:18.142336+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15973,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5425","reported_seq":5547,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.211143+0000","last_change":"2024-04-25T11:45:22.211173+0000","last_active":"2024-04-25T11:45:22.211143+0000","last_peered":"2024-04-25T11:45:22.211143+0000","last_clean":"2024-04-25T11:45:22.211143+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:45:22.211143+0000","last_undegraded":"2024-04-25T11:45:22.211143+0000","last_fullsized":"2024-04-25T11:45:22.211143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5425","last_scrub_stamp":"2024-04-25T11:45:22.211048+0000","last_deep_scrub":"40'5425","last_deep_scrub_stamp":"2024-04-25T11:45:22.211048+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.211048+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:39.029577+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22739,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'9","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.718517+0000","last_change":"2024-04-25T11:45:22.718647+0000","last_active":"2024-04-25T11:45:22.718517+0000","last_peered":"2024-04-25T11:45:22.718517+0000","last_clean":"2024-04-25T11:45:22.718517+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:45:22.718517+0000","last_undegraded":"2024-04-25T11:45:22.718517+0000","last_fullsized":"2024-04-25T11:45:22.718517+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:22.718193+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:22.718193+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.718193+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:10.423674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'13","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.212794+0000","last_change":"2024-04-25T11:45:12.212940+0000","last_active":"2024-04-25T11:45:12.212794+0000","last_peered":"2024-04-25T11:45:12.212794+0000","last_clean":"2024-04-25T11:45:12.212794+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:45:12.212794+0000","last_undegraded":"2024-04-25T11:45:12.212794+0000","last_fullsized":"2024-04-25T11:45:12.212794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:12.212479+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:12.212479+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.212479+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:33.514789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5679","reported_seq":5804,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.682475+0000","last_change":"2024-04-25T11:45:22.682475+0000","last_active":"2024-04-25T11:45:22.682475+0000","last_peered":"2024-04-25T11:45:22.682475+0000","last_clean":"2024-04-25T11:45:22.682475+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:22.682475+0000","last_undegraded":"2024-04-25T11:45:22.682475+0000","last_fullsized":"2024-04-25T11:45:22.682475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5679","last_scrub_stamp":"2024-04-25T11:45:22.682156+0000","last_deep_scrub":"40'5679","last_deep_scrub_stamp":"2024-04-25T11:45:22.682156+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.682156+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:06.150106+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24249,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'17","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.679314+0000","last_change":"2024-04-25T11:45:08.679340+0000","last_active":"2024-04-25T11:45:08.679314+0000","last_peered":"2024-04-25T11:45:08.679314+0000","last_clean":"2024-04-25T11:45:08.679314+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:45:08.679314+0000","last_undegraded":"2024-04-25T11:45:08.679314+0000","last_fullsized":"2024-04-25T11:45:08.679314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.679230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:24.690578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"40'8","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.760031+0000","last_change":"2024-04-25T11:45:15.760136+0000","last_active":"2024-04-25T11:45:15.760031+0000","last_peered":"2024-04-25T11:45:15.760031+0000","last_clean":"2024-04-25T11:45:15.760031+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:45:15.760031+0000","last_undegraded":"2024-04-25T11:45:15.760031+0000","last_fullsized":"2024-04-25T11:45:15.760031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:15.759838+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:15.759838+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.759838+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:44.280027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"40'18","reported_seq":103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.381983+0000","last_change":"2024-04-25T11:45:12.381983+0000","last_active":"2024-04-25T11:45:12.381983+0000","last_peered":"2024-04-25T11:45:12.381983+0000","last_clean":"2024-04-25T11:45:12.381983+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:45:12.381983+0000","last_undegraded":"2024-04-25T11:45:12.381983+0000","last_fullsized":"2024-04-25T11:45:12.381983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:12.381594+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:12.381594+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.381594+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:45.697934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.054174+0000","last_change":"2024-04-25T11:45:13.054174+0000","last_active":"2024-04-25T11:45:13.054174+0000","last_peered":"2024-04-25T11:45:13.054174+0000","last_clean":"2024-04-25T11:45:13.054174+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:45:13.054174+0000","last_undegraded":"2024-04-25T11:45:13.054174+0000","last_fullsized":"2024-04-25T11:45:13.054174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.053765+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.053765+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.053765+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:35.568783+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.786224+0000","last_change":"2024-04-25T11:45:13.786224+0000","last_active":"2024-04-25T11:45:13.786224+0000","last_peered":"2024-04-25T11:45:13.786224+0000","last_clean":"2024-04-25T11:45:13.786224+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:13.786224+0000","last_undegraded":"2024-04-25T11:45:13.786224+0000","last_fullsized":"2024-04-25T11:45:13.786224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4277","last_scrub_stamp":"2024-04-25T11:45:13.786072+0000","last_deep_scrub":"40'4277","last_deep_scrub_stamp":"2024-04-25T11:45:13.786072+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.786072+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:44.692918+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'642","reported_seq":726,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.820194+0000","last_change":"2024-04-25T11:45:14.820194+0000","last_active":"2024-04-25T11:45:14.820194+0000","last_peered":"2024-04-25T11:45:14.820194+0000","last_clean":"2024-04-25T11:45:14.820194+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:45:14.820194+0000","last_undegraded":"2024-04-25T11:45:14.820194+0000","last_fullsized":"2024-04-25T11:45:14.820194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'642","last_scrub_stamp":"2024-04-25T11:45:14.819761+0000","last_deep_scrub":"40'642","last_deep_scrub_stamp":"2024-04-25T11:45:14.819761+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.819761+0000","objects_scrubbed":3,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:58.287711+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.259327+0000","last_change":"2024-04-25T11:45:14.259327+0000","last_active":"2024-04-25T11:45:14.259327+0000","last_peered":"2024-04-25T11:45:14.259327+0000","last_clean":"2024-04-25T11:45:14.259327+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:45:14.259327+0000","last_undegraded":"2024-04-25T11:45:14.259327+0000","last_fullsized":"2024-04-25T11:45:14.259327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:14.259027+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:14.259027+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.259027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:21.674888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5271","reported_seq":5384,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.018210+0000","last_change":"2024-04-25T11:45:23.018210+0000","last_active":"2024-04-25T11:45:23.018210+0000","last_peered":"2024-04-25T11:45:23.018210+0000","last_clean":"2024-04-25T11:45:23.018210+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:45:23.018210+0000","last_undegraded":"2024-04-25T11:45:23.018210+0000","last_fullsized":"2024-04-25T11:45:23.018210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5271","last_scrub_stamp":"2024-04-25T11:45:23.018042+0000","last_deep_scrub":"40'5271","last_deep_scrub_stamp":"2024-04-25T11:45:23.018042+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.018042+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:45:55.375581+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22092,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'15","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.487205+0000","last_change":"2024-04-25T11:45:16.487205+0000","last_active":"2024-04-25T11:45:16.487205+0000","last_peered":"2024-04-25T11:45:16.487205+0000","last_clean":"2024-04-25T11:45:16.487205+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:45:16.487205+0000","last_undegraded":"2024-04-25T11:45:16.487205+0000","last_fullsized":"2024-04-25T11:45:16.487205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:16.486833+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:16.486833+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.486833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:39.001851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.663691+0000","last_change":"2024-04-25T11:45:12.663691+0000","last_active":"2024-04-25T11:45:12.663691+0000","last_peered":"2024-04-25T11:45:12.663691+0000","last_clean":"2024-04-25T11:45:12.663691+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:12.663691+0000","last_undegraded":"2024-04-25T11:45:12.663691+0000","last_fullsized":"2024-04-25T11:45:12.663691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:12.663209+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:12.663209+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.663209+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:52.079695+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'18","reported_seq":91,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.107154+0000","last_change":"2024-04-25T11:45:20.107154+0000","last_active":"2024-04-25T11:45:20.107154+0000","last_peered":"2024-04-25T11:45:20.107154+0000","last_clean":"2024-04-25T11:45:20.107154+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:45:20.107154+0000","last_undegraded":"2024-04-25T11:45:20.107154+0000","last_fullsized":"2024-04-25T11:45:20.107154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:20.106865+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:20.106865+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.106865+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:32.328416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'9","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.079311+0000","last_change":"2024-04-25T11:45:18.079311+0000","last_active":"2024-04-25T11:45:18.079311+0000","last_peered":"2024-04-25T11:45:18.079311+0000","last_clean":"2024-04-25T11:45:18.079311+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:45:18.079311+0000","last_undegraded":"2024-04-25T11:45:18.079311+0000","last_fullsized":"2024-04-25T11:45:18.079311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:18.078968+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:18.078968+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.078968+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:01.546919+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'10","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.676165+0000","last_change":"2024-04-25T11:45:10.676231+0000","last_active":"2024-04-25T11:45:10.676165+0000","last_peered":"2024-04-25T11:45:10.676165+0000","last_clean":"2024-04-25T11:45:10.676165+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:45:10.676165+0000","last_undegraded":"2024-04-25T11:45:10.676165+0000","last_fullsized":"2024-04-25T11:45:10.676165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:10.675970+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:10.675970+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.675970+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:33.373767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.989208+0000","last_change":"2024-04-25T11:45:16.989253+0000","last_active":"2024-04-25T11:45:16.989208+0000","last_peered":"2024-04-25T11:45:16.989208+0000","last_clean":"2024-04-25T11:45:16.989208+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:16.989208+0000","last_undegraded":"2024-04-25T11:45:16.989208+0000","last_fullsized":"2024-04-25T11:45:16.989208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:16.989058+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:16.989058+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.989058+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:35.765902+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'12","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.582718+0000","last_change":"2024-04-25T11:45:19.582853+0000","last_active":"2024-04-25T11:45:19.582718+0000","last_peered":"2024-04-25T11:45:19.582718+0000","last_clean":"2024-04-25T11:45:19.582718+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:45:19.582718+0000","last_undegraded":"2024-04-25T11:45:19.582718+0000","last_fullsized":"2024-04-25T11:45:19.582718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:19.582364+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:19.582364+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.582364+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:16.817070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.839653+0000","last_change":"2024-04-25T11:45:17.839732+0000","last_active":"2024-04-25T11:45:17.839653+0000","last_peered":"2024-04-25T11:45:17.839653+0000","last_clean":"2024-04-25T11:45:17.839653+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:45:17.839653+0000","last_undegraded":"2024-04-25T11:45:17.839653+0000","last_fullsized":"2024-04-25T11:45:17.839653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:17.839416+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:17.839416+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.839416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:54.958479+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'12","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.132888+0000","last_change":"2024-04-25T11:45:21.133023+0000","last_active":"2024-04-25T11:45:21.132888+0000","last_peered":"2024-04-25T11:45:21.132888+0000","last_clean":"2024-04-25T11:45:21.132888+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:45:21.132888+0000","last_undegraded":"2024-04-25T11:45:21.132888+0000","last_fullsized":"2024-04-25T11:45:21.132888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.132544+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.132544+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.132544+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:18.076691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.762701+0000","last_change":"2024-04-25T11:45:12.762800+0000","last_active":"2024-04-25T11:45:12.762701+0000","last_peered":"2024-04-25T11:45:12.762701+0000","last_clean":"2024-04-25T11:45:12.762701+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:45:12.762701+0000","last_undegraded":"2024-04-25T11:45:12.762701+0000","last_fullsized":"2024-04-25T11:45:12.762701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:12.762495+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:12.762495+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.762495+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:22.315164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.253003+0000","last_change":"2024-04-25T11:45:09.253079+0000","last_active":"2024-04-25T11:45:09.253003+0000","last_peered":"2024-04-25T11:45:09.253003+0000","last_clean":"2024-04-25T11:45:09.253003+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:45:09.253003+0000","last_undegraded":"2024-04-25T11:45:09.253003+0000","last_fullsized":"2024-04-25T11:45:09.253003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:09.252761+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:09.252761+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.252761+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:48.105783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'17","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.018421+0000","last_change":"2024-04-25T11:45:11.018505+0000","last_active":"2024-04-25T11:45:11.018421+0000","last_peered":"2024-04-25T11:45:11.018421+0000","last_clean":"2024-04-25T11:45:11.018421+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:45:11.018421+0000","last_undegraded":"2024-04-25T11:45:11.018421+0000","last_fullsized":"2024-04-25T11:45:11.018421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.018243+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.018243+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.018243+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.678307+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.512474+0000","last_change":"2024-04-25T11:45:17.512613+0000","last_active":"2024-04-25T11:45:17.512474+0000","last_peered":"2024-04-25T11:45:17.512474+0000","last_clean":"2024-04-25T11:45:17.512474+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:45:17.512474+0000","last_undegraded":"2024-04-25T11:45:17.512474+0000","last_fullsized":"2024-04-25T11:45:17.512474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:17.512177+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:17.512177+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.512177+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:39.522782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'13","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.752332+0000","last_change":"2024-04-25T11:45:21.752385+0000","last_active":"2024-04-25T11:45:21.752332+0000","last_peered":"2024-04-25T11:45:21.752332+0000","last_clean":"2024-04-25T11:45:21.752332+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:45:21.752332+0000","last_undegraded":"2024-04-25T11:45:21.752332+0000","last_fullsized":"2024-04-25T11:45:21.752332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:21.752173+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:21.752173+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.752173+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:38.570702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.605153+0000","last_change":"2024-04-25T11:45:11.605218+0000","last_active":"2024-04-25T11:45:11.605153+0000","last_peered":"2024-04-25T11:45:11.605153+0000","last_clean":"2024-04-25T11:45:11.605153+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:11.605153+0000","last_undegraded":"2024-04-25T11:45:11.605153+0000","last_fullsized":"2024-04-25T11:45:11.605153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.604952+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.604952+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.604952+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:50.401347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.297638+0000","last_change":"2024-04-25T11:45:17.297676+0000","last_active":"2024-04-25T11:45:17.297638+0000","last_peered":"2024-04-25T11:45:17.297638+0000","last_clean":"2024-04-25T11:45:17.297638+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:45:17.297638+0000","last_undegraded":"2024-04-25T11:45:17.297638+0000","last_fullsized":"2024-04-25T11:45:17.297638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:17.297510+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:17.297510+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.297510+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:06.981200+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.125400+0000","last_change":"2024-04-25T11:45:16.125440+0000","last_active":"2024-04-25T11:45:16.125400+0000","last_peered":"2024-04-25T11:45:16.125400+0000","last_clean":"2024-04-25T11:45:16.125400+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:45:16.125400+0000","last_undegraded":"2024-04-25T11:45:16.125400+0000","last_fullsized":"2024-04-25T11:45:16.125400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:16.125305+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:16.125305+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.125305+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:05.094461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'14","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.980915+0000","last_change":"2024-04-25T11:45:14.980986+0000","last_active":"2024-04-25T11:45:14.980915+0000","last_peered":"2024-04-25T11:45:14.980915+0000","last_clean":"2024-04-25T11:45:14.980915+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:45:14.980915+0000","last_undegraded":"2024-04-25T11:45:14.980915+0000","last_fullsized":"2024-04-25T11:45:14.980915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:14.980689+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:14.980689+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.980689+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:37.236029+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'16","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.011310+0000","last_change":"2024-04-25T11:45:15.011379+0000","last_active":"2024-04-25T11:45:15.011310+0000","last_peered":"2024-04-25T11:45:15.011310+0000","last_clean":"2024-04-25T11:45:15.011310+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:45:15.011310+0000","last_undegraded":"2024-04-25T11:45:15.011310+0000","last_fullsized":"2024-04-25T11:45:15.011310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'16","last_scrub_stamp":"2024-04-25T11:45:15.011069+0000","last_deep_scrub":"40'16","last_deep_scrub_stamp":"2024-04-25T11:45:15.011069+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.011069+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:36.701429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'15","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.785880+0000","last_change":"2024-04-25T11:45:09.785939+0000","last_active":"2024-04-25T11:45:09.785880+0000","last_peered":"2024-04-25T11:45:09.785880+0000","last_clean":"2024-04-25T11:45:09.785880+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:45:09.785880+0000","last_undegraded":"2024-04-25T11:45:09.785880+0000","last_fullsized":"2024-04-25T11:45:09.785880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.785718+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.785718+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.785718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:49:06.298289+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'10","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.291058+0000","last_change":"2024-04-25T11:45:11.291123+0000","last_active":"2024-04-25T11:45:11.291058+0000","last_peered":"2024-04-25T11:45:11.291058+0000","last_clean":"2024-04-25T11:45:11.291058+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:45:11.291058+0000","last_undegraded":"2024-04-25T11:45:11.291058+0000","last_fullsized":"2024-04-25T11:45:11.291058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:11.290861+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:11.290861+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.290861+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:12.768770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.620213+0000","last_change":"2024-04-25T11:45:20.620254+0000","last_active":"2024-04-25T11:45:20.620213+0000","last_peered":"2024-04-25T11:45:20.620213+0000","last_clean":"2024-04-25T11:45:20.620213+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:45:20.620213+0000","last_undegraded":"2024-04-25T11:45:20.620213+0000","last_fullsized":"2024-04-25T11:45:20.620213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:20.620052+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:20.620052+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.620052+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:30.172722+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.562635+0000","last_change":"2024-04-25T11:45:21.562708+0000","last_active":"2024-04-25T11:45:21.562635+0000","last_peered":"2024-04-25T11:45:21.562635+0000","last_clean":"2024-04-25T11:45:21.562635+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:45:21.562635+0000","last_undegraded":"2024-04-25T11:45:21.562635+0000","last_fullsized":"2024-04-25T11:45:21.562635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.562409+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.562409+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.562409+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:40.339253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'15","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:01.944968+0000","last_change":"2024-04-25T11:20:20.051507+0000","last_active":"2024-04-25T11:45:01.944968+0000","last_peered":"2024-04-25T11:45:01.944968+0000","last_clean":"2024-04-25T11:45:01.944968+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:45:01.944968+0000","last_undegraded":"2024-04-25T11:45:01.944968+0000","last_fullsized":"2024-04-25T11:45:01.944968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T11:20:18.972570+0000","last_clean_scrub_stamp":"2024-04-25T11:20:18.972570+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'6","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.843379+0000","last_change":"2024-04-25T11:45:21.843483+0000","last_active":"2024-04-25T11:45:21.843379+0000","last_peered":"2024-04-25T11:45:21.843379+0000","last_clean":"2024-04-25T11:45:21.843379+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:45:21.843379+0000","last_undegraded":"2024-04-25T11:45:21.843379+0000","last_fullsized":"2024-04-25T11:45:21.843379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:21.843208+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:21.843208+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.843208+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:25.069478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.701241+0000","last_change":"2024-04-25T11:45:18.701241+0000","last_active":"2024-04-25T11:45:18.701241+0000","last_peered":"2024-04-25T11:45:18.701241+0000","last_clean":"2024-04-25T11:45:18.701241+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:18.701241+0000","last_undegraded":"2024-04-25T11:45:18.701241+0000","last_fullsized":"2024-04-25T11:45:18.701241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:18.700927+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:18.700927+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.700927+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:02.168877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.080502+0000","last_change":"2024-04-25T11:45:18.080502+0000","last_active":"2024-04-25T11:45:18.080502+0000","last_peered":"2024-04-25T11:45:18.080502+0000","last_clean":"2024-04-25T11:45:18.080502+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:45:18.080502+0000","last_undegraded":"2024-04-25T11:45:18.080502+0000","last_fullsized":"2024-04-25T11:45:18.080502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:18.080106+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:18.080106+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.080106+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:21.882829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'11","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.779099+0000","last_change":"2024-04-25T11:45:19.779099+0000","last_active":"2024-04-25T11:45:19.779099+0000","last_peered":"2024-04-25T11:45:19.779099+0000","last_clean":"2024-04-25T11:45:19.779099+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:45:19.779099+0000","last_undegraded":"2024-04-25T11:45:19.779099+0000","last_fullsized":"2024-04-25T11:45:19.779099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:19.778858+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:19.778858+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.778858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:48.299046+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'5","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.201395+0000","last_change":"2024-04-25T11:45:23.201463+0000","last_active":"2024-04-25T11:45:23.201395+0000","last_peered":"2024-04-25T11:45:23.201395+0000","last_clean":"2024-04-25T11:45:23.201395+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:45:23.201395+0000","last_undegraded":"2024-04-25T11:45:23.201395+0000","last_fullsized":"2024-04-25T11:45:23.201395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:23.201077+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:23.201077+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.201077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:18.322418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.025609+0000","last_change":"2024-04-25T11:45:10.025609+0000","last_active":"2024-04-25T11:45:10.025609+0000","last_peered":"2024-04-25T11:45:10.025609+0000","last_clean":"2024-04-25T11:45:10.025609+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:45:10.025609+0000","last_undegraded":"2024-04-25T11:45:10.025609+0000","last_fullsized":"2024-04-25T11:45:10.025609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:10.025413+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:10.025413+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.025413+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:40.912385+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'6","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.076436+0000","last_change":"2024-04-25T11:45:07.076436+0000","last_active":"2024-04-25T11:45:07.076436+0000","last_peered":"2024-04-25T11:45:07.076436+0000","last_clean":"2024-04-25T11:45:07.076436+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:45:07.076436+0000","last_undegraded":"2024-04-25T11:45:07.076436+0000","last_fullsized":"2024-04-25T11:45:07.076436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:07.076217+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:07.076217+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.076217+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:49.299671+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'7","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.780571+0000","last_change":"2024-04-25T11:45:10.780571+0000","last_active":"2024-04-25T11:45:10.780571+0000","last_peered":"2024-04-25T11:45:10.780571+0000","last_clean":"2024-04-25T11:45:10.780571+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:45:10.780571+0000","last_undegraded":"2024-04-25T11:45:10.780571+0000","last_fullsized":"2024-04-25T11:45:10.780571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:10.780432+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:10.780432+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.780432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:49.633454+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654935,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156929,"ondisk_log_size":156929,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3396,"num_write_kb":16253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3229,"ondisk_log_size":3229,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575155,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000184,"kb_used_data":11824,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93371656,"statfs":{"total":96636764160,"available":95612575744,"internally_reserved":0,"allocated":12107776,"data_stored":11840562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1181784,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":93190056,"statfs":{"total":96636764160,"available":95426617344,"internally_reserved":0,"allocated":4669440,"data_stored":4418265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575156,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776304,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93595536,"statfs":{"total":96636764160,"available":95841828864,"internally_reserved":0,"allocated":499712,"data_stored":226431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575157,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1131992,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93239848,"statfs":{"total":96636764160,"available":95477604352,"internally_reserved":0,"allocated":4669440,"data_stored":4418133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575153,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465336,"kb_used_data":8752,"kb_used_omap":1,"kb_used_meta":1456574,"kb_avail":92906504,"statfs":{"total":96636764160,"available":95136260096,"internally_reserved":0,"allocated":8962048,"data_stored":8703072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1491532231},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575154,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857148,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":856574,"kb_avail":93514692,"statfs":{"total":96636764160,"available":95759044608,"internally_reserved":0,"allocated":577536,"data_stored":314314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877132229},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484956,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":1484478,"kb_avail":92886884,"statfs":{"total":96636764160,"available":95116169216,"internally_reserved":0,"allocated":479232,"data_stored":223217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1520105926},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575154,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054448,"kb_used_data":3624,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93317392,"statfs":{"total":96636764160,"available":95557009408,"internally_reserved":0,"allocated":3710976,"data_stored":3449798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:30.853 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-25T11:20:18.972570+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=20, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=11, tm_min=45, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T11:45:30.854 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T11:45:50.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T11:45:51.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.067+0000 7f3f7eb9e640 1 Processor -- start 2024-04-25T11:45:51.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.067+0000 7f3f7eb9e640 1 -- start start 2024-04-25T11:45:51.055 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7eb9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7eb9e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3f7815a430 0x7f3f7815a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7eb9e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f7815d750 con 0x7f3f780b5060 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7eb9e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3f7815da30 con 0x7f3f7815a430 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7eb9e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f7815dd10 con 0x7f3f780ca290 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f6ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7d13b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3f7815a430 0x7f3f7815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7c93a640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3f780b5060 0x7f3f780c4520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.136:35270/0 (socket says 172.21.15.136:35270) 2024-04-25T11:45:51.056 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7c93a640 1 -- 172.21.15.136:0/2835914553 learned_addr learned my addr 172.21.15.136:0/2835914553 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:51.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7d13b640 1 -- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 msgr2=0x7f3f780ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7d13b640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7d13b640 1 -- 172.21.15.136:0/2835914553 >> v1:172.21.15.157:6789/0 conn(0x7f3f780b5060 legacy=0x7f3f780c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f7d13b640 1 -- 172.21.15.136:0/2835914553 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7815dff0 con 0x7f3f7815a430 2024-04-25T11:45:51.057 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.071+0000 7f3f6ffff640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T11:45:51.059 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7d13b640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f7815a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009d90 tx=0x7f3f7001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de04a5220ddbd7b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f6f7fe640 1 -- 172.21.15.136:0/2835914553 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700b4070 con 0x7f3f7815a430 2024-04-25T11:45:51.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f6f7fe640 1 -- 172.21.15.136:0/2835914553 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f700af600 con 0x7f3f7815a430 2024-04-25T11:45:51.060 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f6f7fe640 1 -- 172.21.15.136:0/2835914553 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f700af9d0 con 0x7f3f7815a430 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 msgr2=0x7f3f7815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f7815a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009d90 tx=0x7f3f7001f150 comp rx=0 tx=0).stop 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/2835914553 shutdown_connections 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f7815a840 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/2835914553 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/2835914553 >> 172.21.15.136:0/2835914553 conn(0x7f3f78070f70 msgr2=0x7f3f78068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/2835914553 shutdown_connections 2024-04-25T11:45:51.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.075+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/2835914553 wait complete. 2024-04-25T11:45:51.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 Processor -- start 2024-04-25T11:45:51.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- start start 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f78147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7815ad80 con 0x7f3f780b5060 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78159f10 con 0x7f3f7815a430 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f7815d3f0 con 0x7f3f780ca290 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38486/0 (socket says 172.21.15.136:38486) 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 -- 172.21.15.136:0/3459820308 learned_addr learned my addr 172.21.15.136:0/3459820308 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f6ffff640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7d13b640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f78147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 -- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 msgr2=0x7f3f780cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 -- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 msgr2=0x7f3f78147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f78147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 -- 172.21.15.136:0/3459820308 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f780b2090 con 0x7f3f780b5060 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f6ffff640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7c93a640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800c400 tx=0x7f3f6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68017070 con 0x7f3f780b5060 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f68005cf0 con 0x7f3f780b5060 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7806f790 con 0x7f3f780b5060 2024-04-25T11:45:51.066 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680072d0 con 0x7f3f780b5060 2024-04-25T11:45:51.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f78157ce0 con 0x7f3f780b5060 2024-04-25T11:45:51.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.079+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f780b2260 con 0x7f3f780b5060 2024-04-25T11:45:51.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.083+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3f6801a070 con 0x7f3f780b5060 2024-04-25T11:45:51.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.083+0000 7f3f6d7fa640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 0x7f3f64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.067 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.083+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f3f6808ee40 con 0x7f3f780b5060 2024-04-25T11:45:51.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.083+0000 7f3f6ffff640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 0x7f3f64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.068 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.083+0000 7f3f6ffff640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 0x7f3f64071940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f781507a0 tx=0x7f3f74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.072 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.087+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3f6805ae00 con 0x7f3f780b5060 2024-04-25T11:45:51.230 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.243+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 --> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f7806f260 con 0x7f3f6406f430 2024-04-25T11:45:51.254 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.267+0000 7f3f6d7fa640 1 -- 172.21.15.136:0/3459820308 <== mgr.4102 v2:172.21.15.157:6832/602770275 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+334643 (secure 0 0 0) 0x7f3f7806f260 con 0x7f3f6406f430 2024-04-25T11:45:51.258 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:51.260 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.271+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 msgr2=0x7f3f64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 0x7f3f64071940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f781507a0 tx=0x7f3f74008040 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 msgr2=0x7f3f780b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800c400 tx=0x7f3f6800c8c0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 shutdown_connections 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f7815a430 0x7f3f78147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3f6406f430 0x7f3f64071940 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f780ca290 0x7f3f780cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 --2- 172.21.15.136:0/3459820308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f780b5060 0x7f3f780b1b50 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 >> 172.21.15.136:0/3459820308 conn(0x7f3f78070f70 msgr2=0x7f3f780ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:51.261 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 shutdown_connections 2024-04-25T11:45:51.262 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.275+0000 7f3f7eb9e640 1 -- 172.21.15.136:0/3459820308 wait complete. 2024-04-25T11:45:51.277 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-04-25T11:45:49.439278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15910156,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2584,"num_read_kb":33411,"num_write":658432,"num_write_kb":15736463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160971,"ondisk_log_size":160971,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8952152,"kb_used_data":34840,"kb_used_omap":12,"kb_used_meta":8917235,"kb_avail":746022568,"statfs":{"total":773094113280,"available":763927109632,"internally_reserved":0,"allocated":35676160,"data_stored":33593792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":9131249202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006682"},"pg_stats":[{"pgid":"2.1f","version":"40'283","reported_seq":379,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.960822+0000","last_change":"2024-04-25T11:45:08.960822+0000","last_active":"2024-04-25T11:45:08.960822+0000","last_peered":"2024-04-25T11:45:08.960822+0000","last_clean":"2024-04-25T11:45:08.960822+0000","last_became_active":"2024-04-25T11:20:19.031588+0000","last_became_peered":"2024-04-25T11:20:19.031588+0000","last_unstale":"2024-04-25T11:45:08.960822+0000","last_undegraded":"2024-04-25T11:45:08.960822+0000","last_fullsized":"2024-04-25T11:45:08.960822+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'283","last_scrub_stamp":"2024-04-25T11:45:08.960541+0000","last_deep_scrub":"40'283","last_deep_scrub_stamp":"2024-04-25T11:45:08.960541+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.960541+0000","objects_scrubbed":7,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:28.785626+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":264,"num_write_kb":2720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"40'7","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.851320+0000","last_change":"2024-04-25T11:45:16.851449+0000","last_active":"2024-04-25T11:45:16.851320+0000","last_peered":"2024-04-25T11:45:16.851320+0000","last_clean":"2024-04-25T11:45:16.851320+0000","last_became_active":"2024-04-25T11:20:20.023067+0000","last_became_peered":"2024-04-25T11:20:20.023067+0000","last_unstale":"2024-04-25T11:45:16.851320+0000","last_undegraded":"2024-04-25T11:45:16.851320+0000","last_fullsized":"2024-04-25T11:45:16.851320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:16.851008+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:16.851008+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.851008+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:27.594650+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"40'5345","reported_seq":5459,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:27.899345+0000","last_change":"2024-04-25T11:45:27.899377+0000","last_active":"2024-04-25T11:45:27.899345+0000","last_peered":"2024-04-25T11:45:27.899345+0000","last_clean":"2024-04-25T11:45:27.899345+0000","last_became_active":"2024-04-25T11:20:24.165233+0000","last_became_peered":"2024-04-25T11:20:24.165233+0000","last_unstale":"2024-04-25T11:45:27.899345+0000","last_undegraded":"2024-04-25T11:45:27.899345+0000","last_fullsized":"2024-04-25T11:45:27.899345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5345","last_scrub_stamp":"2024-04-25T11:45:27.899232+0000","last_deep_scrub":"40'5345","last_deep_scrub_stamp":"2024-04-25T11:45:27.899232+0000","last_clean_scrub_stamp":"2024-04-25T11:45:27.899232+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:39.688345+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":22060,"num_write_kb":542611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"40'11","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.792774+0000","last_change":"2024-04-25T11:45:13.792841+0000","last_active":"2024-04-25T11:45:13.792774+0000","last_peered":"2024-04-25T11:45:13.792774+0000","last_clean":"2024-04-25T11:45:13.792774+0000","last_became_active":"2024-04-25T11:20:19.003493+0000","last_became_peered":"2024-04-25T11:20:19.003493+0000","last_unstale":"2024-04-25T11:45:13.792774+0000","last_undegraded":"2024-04-25T11:45:13.792774+0000","last_fullsized":"2024-04-25T11:45:13.792774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.792577+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.792577+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.792577+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:32.569421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.237635+0000","last_change":"2024-04-25T11:45:17.237711+0000","last_active":"2024-04-25T11:45:17.237635+0000","last_peered":"2024-04-25T11:45:17.237635+0000","last_clean":"2024-04-25T11:45:17.237635+0000","last_became_active":"2024-04-25T11:20:20.034968+0000","last_became_peered":"2024-04-25T11:20:20.034968+0000","last_unstale":"2024-04-25T11:45:17.237635+0000","last_undegraded":"2024-04-25T11:45:17.237635+0000","last_fullsized":"2024-04-25T11:45:17.237635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:17.237394+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:17.237394+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.237394+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:36.014238+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"40'4856","reported_seq":4969,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.440747+0000","last_change":"2024-04-25T11:45:19.440776+0000","last_active":"2024-04-25T11:45:19.440747+0000","last_peered":"2024-04-25T11:45:19.440747+0000","last_clean":"2024-04-25T11:45:19.440747+0000","last_became_active":"2024-04-25T11:20:24.093786+0000","last_became_peered":"2024-04-25T11:20:24.093786+0000","last_unstale":"2024-04-25T11:45:19.440747+0000","last_undegraded":"2024-04-25T11:45:19.440747+0000","last_fullsized":"2024-04-25T11:45:19.440747+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4856","last_scrub_stamp":"2024-04-25T11:45:19.440645+0000","last_deep_scrub":"40'4856","last_deep_scrub_stamp":"2024-04-25T11:45:19.440645+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.440645+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:28.993315+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":20164,"num_write_kb":506205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"40'37","reported_seq":141,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.702835+0000","last_change":"2024-04-25T11:45:15.702835+0000","last_active":"2024-04-25T11:45:15.702835+0000","last_peered":"2024-04-25T11:45:15.702835+0000","last_clean":"2024-04-25T11:45:15.702835+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:45:15.702835+0000","last_undegraded":"2024-04-25T11:45:15.702835+0000","last_fullsized":"2024-04-25T11:45:15.702835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'37","last_scrub_stamp":"2024-04-25T11:45:15.702508+0000","last_deep_scrub":"40'37","last_deep_scrub_stamp":"2024-04-25T11:45:15.702508+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.702508+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:37.392868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":87,"num_read_kb":95,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"40'7","reported_seq":63,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.636572+0000","last_change":"2024-04-25T11:45:14.636646+0000","last_active":"2024-04-25T11:45:14.636572+0000","last_peered":"2024-04-25T11:45:14.636572+0000","last_clean":"2024-04-25T11:45:14.636572+0000","last_became_active":"2024-04-25T11:21:00.072527+0000","last_became_peered":"2024-04-25T11:21:00.072527+0000","last_unstale":"2024-04-25T11:45:14.636572+0000","last_undegraded":"2024-04-25T11:45:14.636572+0000","last_fullsized":"2024-04-25T11:45:14.636572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:14.636335+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:14.636335+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.636335+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:40.225075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"40'4265","reported_seq":4370,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.354003+0000","last_change":"2024-04-25T11:45:20.354003+0000","last_active":"2024-04-25T11:45:20.354003+0000","last_peered":"2024-04-25T11:45:20.354003+0000","last_clean":"2024-04-25T11:45:20.354003+0000","last_became_active":"2024-04-25T11:20:24.097398+0000","last_became_peered":"2024-04-25T11:20:24.097398+0000","last_unstale":"2024-04-25T11:45:20.354003+0000","last_undegraded":"2024-04-25T11:45:20.354003+0000","last_fullsized":"2024-04-25T11:45:20.354003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4265","last_scrub_stamp":"2024-04-25T11:45:20.353862+0000","last_deep_scrub":"40'4265","last_deep_scrub_stamp":"2024-04-25T11:45:20.353862+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.353862+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:39.130706+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17222,"num_write_kb":433214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"40'11","reported_seq":102,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.736889+0000","last_change":"2024-04-25T11:45:17.736889+0000","last_active":"2024-04-25T11:45:17.736889+0000","last_peered":"2024-04-25T11:45:17.736889+0000","last_clean":"2024-04-25T11:45:17.736889+0000","last_became_active":"2024-04-25T11:20:19.046711+0000","last_became_peered":"2024-04-25T11:20:19.046711+0000","last_unstale":"2024-04-25T11:45:17.736889+0000","last_undegraded":"2024-04-25T11:45:17.736889+0000","last_fullsized":"2024-04-25T11:45:17.736889+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:17.736473+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:17.736473+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.736473+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:53.767593+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"40'15","reported_seq":71,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.631676+0000","last_change":"2024-04-25T11:45:15.631751+0000","last_active":"2024-04-25T11:45:15.631676+0000","last_peered":"2024-04-25T11:45:15.631676+0000","last_clean":"2024-04-25T11:45:15.631676+0000","last_became_active":"2024-04-25T11:21:00.072319+0000","last_became_peered":"2024-04-25T11:21:00.072319+0000","last_unstale":"2024-04-25T11:45:15.631676+0000","last_undegraded":"2024-04-25T11:45:15.631676+0000","last_fullsized":"2024-04-25T11:45:15.631676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:15.631432+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:15.631432+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.631432+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:57.854886+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"40'4169","reported_seq":4282,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.500583+0000","last_change":"2024-04-25T11:45:15.500583+0000","last_active":"2024-04-25T11:45:15.500583+0000","last_peered":"2024-04-25T11:45:15.500583+0000","last_clean":"2024-04-25T11:45:15.500583+0000","last_became_active":"2024-04-25T11:20:24.092237+0000","last_became_peered":"2024-04-25T11:20:24.092237+0000","last_unstale":"2024-04-25T11:45:15.500583+0000","last_undegraded":"2024-04-25T11:45:15.500583+0000","last_fullsized":"2024-04-25T11:45:15.500583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4169","last_scrub_stamp":"2024-04-25T11:45:15.500247+0000","last_deep_scrub":"40'4169","last_deep_scrub_stamp":"2024-04-25T11:45:15.500247+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.500247+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.015266+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":16708,"num_write_kb":435074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"40'10","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.553565+0000","last_change":"2024-04-25T11:45:18.553627+0000","last_active":"2024-04-25T11:45:18.553565+0000","last_peered":"2024-04-25T11:45:18.553565+0000","last_clean":"2024-04-25T11:45:18.553565+0000","last_became_active":"2024-04-25T11:20:20.017777+0000","last_became_peered":"2024-04-25T11:20:20.017777+0000","last_unstale":"2024-04-25T11:45:18.553565+0000","last_undegraded":"2024-04-25T11:45:18.553565+0000","last_fullsized":"2024-04-25T11:45:18.553565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:18.553383+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:18.553383+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.553383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:41.251180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"40'1093","reported_seq":1182,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.734685+0000","last_change":"2024-04-25T11:45:16.734777+0000","last_active":"2024-04-25T11:45:16.734685+0000","last_peered":"2024-04-25T11:45:16.734685+0000","last_clean":"2024-04-25T11:45:16.734685+0000","last_became_active":"2024-04-25T11:20:19.042778+0000","last_became_peered":"2024-04-25T11:20:19.042778+0000","last_unstale":"2024-04-25T11:45:16.734685+0000","last_undegraded":"2024-04-25T11:45:16.734685+0000","last_fullsized":"2024-04-25T11:45:16.734685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'1093","last_scrub_stamp":"2024-04-25T11:45:16.734476+0000","last_deep_scrub":"40'1093","last_deep_scrub_stamp":"2024-04-25T11:45:16.734476+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.734476+0000","objects_scrubbed":3,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:03.233019+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":7610,"num_write":1081,"num_write_kb":4411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"40'5159","reported_seq":5267,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.480064+0000","last_change":"2024-04-25T11:45:16.480096+0000","last_active":"2024-04-25T11:45:16.480064+0000","last_peered":"2024-04-25T11:45:16.480064+0000","last_clean":"2024-04-25T11:45:16.480064+0000","last_became_active":"2024-04-25T11:20:24.092001+0000","last_became_peered":"2024-04-25T11:20:24.092001+0000","last_unstale":"2024-04-25T11:45:16.480064+0000","last_undegraded":"2024-04-25T11:45:16.480064+0000","last_fullsized":"2024-04-25T11:45:16.480064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5159","last_scrub_stamp":"2024-04-25T11:45:16.479962+0000","last_deep_scrub":"40'5159","last_deep_scrub_stamp":"2024-04-25T11:45:16.479962+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.479962+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:38.490567+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21468,"num_write_kb":532122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"40'9","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.837718+0000","last_change":"2024-04-25T11:45:19.837792+0000","last_active":"2024-04-25T11:45:19.837718+0000","last_peered":"2024-04-25T11:45:19.837718+0000","last_clean":"2024-04-25T11:45:19.837718+0000","last_became_active":"2024-04-25T11:20:20.034255+0000","last_became_peered":"2024-04-25T11:20:20.034255+0000","last_unstale":"2024-04-25T11:45:19.837718+0000","last_undegraded":"2024-04-25T11:45:19.837718+0000","last_fullsized":"2024-04-25T11:45:19.837718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:19.837486+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:19.837486+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.837486+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:07.542967+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"40'15","reported_seq":94,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.042850+0000","last_change":"2024-04-25T11:45:09.042888+0000","last_active":"2024-04-25T11:45:09.042850+0000","last_peered":"2024-04-25T11:45:09.042850+0000","last_clean":"2024-04-25T11:45:09.042850+0000","last_became_active":"2024-04-25T11:20:19.000992+0000","last_became_peered":"2024-04-25T11:20:19.000992+0000","last_unstale":"2024-04-25T11:45:09.042850+0000","last_undegraded":"2024-04-25T11:45:09.042850+0000","last_fullsized":"2024-04-25T11:45:09.042850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.042755+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.042755+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.042755+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:48:28.867840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"40'5171","reported_seq":5278,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.399544+0000","last_change":"2024-04-25T11:45:22.399653+0000","last_active":"2024-04-25T11:45:22.399544+0000","last_peered":"2024-04-25T11:45:22.399544+0000","last_clean":"2024-04-25T11:45:22.399544+0000","last_became_active":"2024-04-25T11:20:24.097848+0000","last_became_peered":"2024-04-25T11:20:24.097848+0000","last_unstale":"2024-04-25T11:45:22.399544+0000","last_undegraded":"2024-04-25T11:45:22.399544+0000","last_fullsized":"2024-04-25T11:45:22.399544+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5171","last_scrub_stamp":"2024-04-25T11:45:22.399259+0000","last_deep_scrub":"40'5171","last_deep_scrub_stamp":"2024-04-25T11:45:22.399259+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.399259+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.919447+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":21566,"num_write_kb":475272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"40'9","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.747264+0000","last_change":"2024-04-25T11:45:18.747314+0000","last_active":"2024-04-25T11:45:18.747264+0000","last_peered":"2024-04-25T11:45:18.747264+0000","last_clean":"2024-04-25T11:45:18.747264+0000","last_became_active":"2024-04-25T11:20:20.034194+0000","last_became_peered":"2024-04-25T11:20:20.034194+0000","last_unstale":"2024-04-25T11:45:18.747264+0000","last_undegraded":"2024-04-25T11:45:18.747264+0000","last_fullsized":"2024-04-25T11:45:18.747264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:18.747128+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:18.747128+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.747128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:35.565657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"40'13","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.953773+0000","last_change":"2024-04-25T11:45:10.953852+0000","last_active":"2024-04-25T11:45:10.953773+0000","last_peered":"2024-04-25T11:45:10.953773+0000","last_clean":"2024-04-25T11:45:10.953773+0000","last_became_active":"2024-04-25T11:20:19.031878+0000","last_became_peered":"2024-04-25T11:20:19.031878+0000","last_unstale":"2024-04-25T11:45:10.953773+0000","last_undegraded":"2024-04-25T11:45:10.953773+0000","last_fullsized":"2024-04-25T11:45:10.953773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:10.953598+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:10.953598+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.953598+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.613663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"40'6596","reported_seq":6717,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.840640+0000","last_change":"2024-04-25T11:45:16.840671+0000","last_active":"2024-04-25T11:45:16.840640+0000","last_peered":"2024-04-25T11:45:16.840640+0000","last_clean":"2024-04-25T11:45:16.840640+0000","last_became_active":"2024-04-25T11:21:00.112475+0000","last_became_peered":"2024-04-25T11:21:00.112475+0000","last_unstale":"2024-04-25T11:45:16.840640+0000","last_undegraded":"2024-04-25T11:45:16.840640+0000","last_fullsized":"2024-04-25T11:45:16.840640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6596","last_scrub_stamp":"2024-04-25T11:45:16.840533+0000","last_deep_scrub":"40'6596","last_deep_scrub_stamp":"2024-04-25T11:45:16.840533+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.840533+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:18.318443+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":0,"num_write":27527,"num_write_kb":646275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.333845+0000","last_change":"2024-04-25T11:45:10.333911+0000","last_active":"2024-04-25T11:45:10.333845+0000","last_peered":"2024-04-25T11:45:10.333845+0000","last_clean":"2024-04-25T11:45:10.333845+0000","last_became_active":"2024-04-25T11:20:20.006800+0000","last_became_peered":"2024-04-25T11:20:20.006800+0000","last_unstale":"2024-04-25T11:45:10.333845+0000","last_undegraded":"2024-04-25T11:45:10.333845+0000","last_fullsized":"2024-04-25T11:45:10.333845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:10.333643+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:10.333643+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.333643+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:52.557097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"40'8","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.646042+0000","last_change":"2024-04-25T11:45:12.646193+0000","last_active":"2024-04-25T11:45:12.646042+0000","last_peered":"2024-04-25T11:45:12.646042+0000","last_clean":"2024-04-25T11:45:12.646042+0000","last_became_active":"2024-04-25T11:20:19.032375+0000","last_became_peered":"2024-04-25T11:20:19.032375+0000","last_unstale":"2024-04-25T11:45:12.646042+0000","last_undegraded":"2024-04-25T11:45:12.646042+0000","last_fullsized":"2024-04-25T11:45:12.646042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:12.645732+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:12.645732+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.645732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:28.657080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"40'4795","reported_seq":4901,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.515960+0000","last_change":"2024-04-25T11:45:17.515990+0000","last_active":"2024-04-25T11:45:17.515960+0000","last_peered":"2024-04-25T11:45:17.515960+0000","last_clean":"2024-04-25T11:45:17.515960+0000","last_became_active":"2024-04-25T11:20:24.094371+0000","last_became_peered":"2024-04-25T11:20:24.094371+0000","last_unstale":"2024-04-25T11:45:17.515960+0000","last_undegraded":"2024-04-25T11:45:17.515960+0000","last_fullsized":"2024-04-25T11:45:17.515960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4795","last_scrub_stamp":"2024-04-25T11:45:17.515869+0000","last_deep_scrub":"40'4795","last_deep_scrub_stamp":"2024-04-25T11:45:17.515869+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.515869+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:40.213666+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":19675,"num_write_kb":467661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"40'18","reported_seq":89,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:06.965535+0000","last_change":"2024-04-25T11:45:06.965581+0000","last_active":"2024-04-25T11:45:06.965535+0000","last_peered":"2024-04-25T11:45:06.965535+0000","last_clean":"2024-04-25T11:45:06.965535+0000","last_became_active":"2024-04-25T11:20:20.031772+0000","last_became_peered":"2024-04-25T11:20:20.031772+0000","last_unstale":"2024-04-25T11:45:06.965535+0000","last_undegraded":"2024-04-25T11:45:06.965535+0000","last_fullsized":"2024-04-25T11:45:06.965535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:06.965415+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:06.965415+0000","last_clean_scrub_stamp":"2024-04-25T11:45:06.965415+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:27:21.084447+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"40'18","reported_seq":114,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.407459+0000","last_change":"2024-04-25T11:45:13.407575+0000","last_active":"2024-04-25T11:45:13.407459+0000","last_peered":"2024-04-25T11:45:13.407459+0000","last_clean":"2024-04-25T11:45:13.407459+0000","last_became_active":"2024-04-25T11:20:19.038962+0000","last_became_peered":"2024-04-25T11:20:19.038962+0000","last_unstale":"2024-04-25T11:45:13.407459+0000","last_undegraded":"2024-04-25T11:45:13.407459+0000","last_fullsized":"2024-04-25T11:45:13.407459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:13.407199+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:13.407199+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.407199+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:12.797988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":15,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"40'4482","reported_seq":4593,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.385961+0000","last_change":"2024-04-25T11:45:23.385961+0000","last_active":"2024-04-25T11:45:23.385961+0000","last_peered":"2024-04-25T11:45:23.385961+0000","last_clean":"2024-04-25T11:45:23.385961+0000","last_became_active":"2024-04-25T11:20:24.166392+0000","last_became_peered":"2024-04-25T11:20:24.166392+0000","last_unstale":"2024-04-25T11:45:23.385961+0000","last_undegraded":"2024-04-25T11:45:23.385961+0000","last_fullsized":"2024-04-25T11:45:23.385961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4482","last_scrub_stamp":"2024-04-25T11:45:23.385828+0000","last_deep_scrub":"40'4482","last_deep_scrub_stamp":"2024-04-25T11:45:23.385828+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.385828+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:34.129470+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":18513,"num_write_kb":466799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.880339+0000","last_change":"2024-04-25T11:45:20.880417+0000","last_active":"2024-04-25T11:45:20.880339+0000","last_peered":"2024-04-25T11:45:20.880339+0000","last_clean":"2024-04-25T11:45:20.880339+0000","last_became_active":"2024-04-25T11:20:20.035174+0000","last_became_peered":"2024-04-25T11:20:20.035174+0000","last_unstale":"2024-04-25T11:45:20.880339+0000","last_undegraded":"2024-04-25T11:45:20.880339+0000","last_fullsized":"2024-04-25T11:45:20.880339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:20.880111+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:20.880111+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.880111+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:08.095052+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"40'372","reported_seq":461,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.936453+0000","last_change":"2024-04-25T11:45:11.936609+0000","last_active":"2024-04-25T11:45:11.936453+0000","last_peered":"2024-04-25T11:45:11.936453+0000","last_clean":"2024-04-25T11:45:11.936453+0000","last_became_active":"2024-04-25T11:20:19.035320+0000","last_became_peered":"2024-04-25T11:20:19.035320+0000","last_unstale":"2024-04-25T11:45:11.936453+0000","last_undegraded":"2024-04-25T11:45:11.936453+0000","last_fullsized":"2024-04-25T11:45:11.936453+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'372","last_scrub_stamp":"2024-04-25T11:45:11.936219+0000","last_deep_scrub":"40'372","last_deep_scrub_stamp":"2024-04-25T11:45:11.936219+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.936219+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:11.984038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":784,"num_write_kb":279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"40'4559","reported_seq":4668,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:26.915942+0000","last_change":"2024-04-25T11:45:26.916023+0000","last_active":"2024-04-25T11:45:26.915942+0000","last_peered":"2024-04-25T11:45:26.915942+0000","last_clean":"2024-04-25T11:45:26.915942+0000","last_became_active":"2024-04-25T11:20:24.161952+0000","last_became_peered":"2024-04-25T11:20:24.161952+0000","last_unstale":"2024-04-25T11:45:26.915942+0000","last_undegraded":"2024-04-25T11:45:26.915942+0000","last_fullsized":"2024-04-25T11:45:26.915942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4559","last_scrub_stamp":"2024-04-25T11:45:26.915685+0000","last_deep_scrub":"40'4559","last_deep_scrub_stamp":"2024-04-25T11:45:26.915685+0000","last_clean_scrub_stamp":"2024-04-25T11:45:26.915685+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:55.236848+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":19035,"num_write_kb":452673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"40'9","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.906560+0000","last_change":"2024-04-25T11:45:15.906560+0000","last_active":"2024-04-25T11:45:15.906560+0000","last_peered":"2024-04-25T11:45:15.906560+0000","last_clean":"2024-04-25T11:45:15.906560+0000","last_became_active":"2024-04-25T11:20:20.031410+0000","last_became_peered":"2024-04-25T11:20:20.031410+0000","last_unstale":"2024-04-25T11:45:15.906560+0000","last_undegraded":"2024-04-25T11:45:15.906560+0000","last_fullsized":"2024-04-25T11:45:15.906560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:15.906213+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:15.906213+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.906213+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:02.404755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"40'9","reported_seq":95,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.709994+0000","last_change":"2024-04-25T11:45:14.709994+0000","last_active":"2024-04-25T11:45:14.709994+0000","last_peered":"2024-04-25T11:45:14.709994+0000","last_clean":"2024-04-25T11:45:14.709994+0000","last_became_active":"2024-04-25T11:20:19.041945+0000","last_became_peered":"2024-04-25T11:20:19.041945+0000","last_unstale":"2024-04-25T11:45:14.709994+0000","last_undegraded":"2024-04-25T11:45:14.709994+0000","last_fullsized":"2024-04-25T11:45:14.709994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:14.709707+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:14.709707+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.709707+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:36.012017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":8,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"40'5085","reported_seq":5196,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.227318+0000","last_change":"2024-04-25T11:45:21.227318+0000","last_active":"2024-04-25T11:45:21.227318+0000","last_peered":"2024-04-25T11:45:21.227318+0000","last_clean":"2024-04-25T11:45:21.227318+0000","last_became_active":"2024-04-25T11:20:24.161129+0000","last_became_peered":"2024-04-25T11:20:24.161129+0000","last_unstale":"2024-04-25T11:45:21.227318+0000","last_undegraded":"2024-04-25T11:45:21.227318+0000","last_fullsized":"2024-04-25T11:45:21.227318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5085","last_scrub_stamp":"2024-04-25T11:45:21.227103+0000","last_deep_scrub":"40'5085","last_deep_scrub_stamp":"2024-04-25T11:45:21.227103+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.227103+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:58.346166+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":0,"num_write":21093,"num_write_kb":505918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"40'14","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.186717+0000","last_change":"2024-04-25T11:45:22.186717+0000","last_active":"2024-04-25T11:45:22.186717+0000","last_peered":"2024-04-25T11:45:22.186717+0000","last_clean":"2024-04-25T11:45:22.186717+0000","last_became_active":"2024-04-25T11:20:20.039886+0000","last_became_peered":"2024-04-25T11:20:20.039886+0000","last_unstale":"2024-04-25T11:45:22.186717+0000","last_undegraded":"2024-04-25T11:45:22.186717+0000","last_fullsized":"2024-04-25T11:45:22.186717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:22.186318+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:22.186318+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.186318+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:09.891799+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"40'5","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.037323+0000","last_change":"2024-04-25T11:45:08.037323+0000","last_active":"2024-04-25T11:45:08.037323+0000","last_peered":"2024-04-25T11:45:08.037323+0000","last_clean":"2024-04-25T11:45:08.037323+0000","last_became_active":"2024-04-25T11:20:18.998977+0000","last_became_peered":"2024-04-25T11:20:18.998977+0000","last_unstale":"2024-04-25T11:45:08.037323+0000","last_undegraded":"2024-04-25T11:45:08.037323+0000","last_fullsized":"2024-04-25T11:45:08.037323+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:08.037222+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:08.037222+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.037222+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:41.353562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"40'5304","reported_seq":5420,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.914766+0000","last_change":"2024-04-25T11:45:20.914766+0000","last_active":"2024-04-25T11:45:20.914766+0000","last_peered":"2024-04-25T11:45:20.914766+0000","last_clean":"2024-04-25T11:45:20.914766+0000","last_became_active":"2024-04-25T11:20:24.092789+0000","last_became_peered":"2024-04-25T11:20:24.092789+0000","last_unstale":"2024-04-25T11:45:20.914766+0000","last_undegraded":"2024-04-25T11:45:20.914766+0000","last_fullsized":"2024-04-25T11:45:20.914766+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5304","last_scrub_stamp":"2024-04-25T11:45:20.914252+0000","last_deep_scrub":"40'5304","last_deep_scrub_stamp":"2024-04-25T11:45:20.914252+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.914252+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:58.033315+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21947,"num_write_kb":517667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"40'9","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.289546+0000","last_change":"2024-04-25T11:45:12.289546+0000","last_active":"2024-04-25T11:45:12.289546+0000","last_peered":"2024-04-25T11:45:12.289546+0000","last_clean":"2024-04-25T11:45:12.289546+0000","last_became_active":"2024-04-25T11:20:20.012491+0000","last_became_peered":"2024-04-25T11:20:20.012491+0000","last_unstale":"2024-04-25T11:45:12.289546+0000","last_undegraded":"2024-04-25T11:45:12.289546+0000","last_fullsized":"2024-04-25T11:45:12.289546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:12.289275+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:12.289275+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.289275+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:59.622600+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"40'17","reported_seq":108,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.212252+0000","last_change":"2024-04-25T11:45:11.212252+0000","last_active":"2024-04-25T11:45:11.212252+0000","last_peered":"2024-04-25T11:45:11.212252+0000","last_clean":"2024-04-25T11:45:11.212252+0000","last_became_active":"2024-04-25T11:20:19.021709+0000","last_became_peered":"2024-04-25T11:20:19.021709+0000","last_unstale":"2024-04-25T11:45:11.212252+0000","last_undegraded":"2024-04-25T11:45:11.212252+0000","last_fullsized":"2024-04-25T11:45:11.212252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.211966+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.211966+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.211966+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:33.726984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"40'5442","reported_seq":5562,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.959686+0000","last_change":"2024-04-25T11:45:19.959686+0000","last_active":"2024-04-25T11:45:19.959686+0000","last_peered":"2024-04-25T11:45:19.959686+0000","last_clean":"2024-04-25T11:45:19.959686+0000","last_became_active":"2024-04-25T11:20:24.084855+0000","last_became_peered":"2024-04-25T11:20:24.084855+0000","last_unstale":"2024-04-25T11:45:19.959686+0000","last_undegraded":"2024-04-25T11:45:19.959686+0000","last_fullsized":"2024-04-25T11:45:19.959686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5442","last_scrub_stamp":"2024-04-25T11:45:19.959160+0000","last_deep_scrub":"40'5442","last_deep_scrub_stamp":"2024-04-25T11:45:19.959160+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.959160+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:30.702802+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":0,"num_write":23011,"num_write_kb":550608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"40'12","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.903319+0000","last_change":"2024-04-25T11:45:13.903450+0000","last_active":"2024-04-25T11:45:13.903319+0000","last_peered":"2024-04-25T11:45:13.903319+0000","last_clean":"2024-04-25T11:45:13.903319+0000","last_became_active":"2024-04-25T11:20:20.022956+0000","last_became_peered":"2024-04-25T11:20:20.022956+0000","last_unstale":"2024-04-25T11:45:13.903319+0000","last_undegraded":"2024-04-25T11:45:13.903319+0000","last_fullsized":"2024-04-25T11:45:13.903319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:13.903084+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:13.903084+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.903084+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:35.913689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"40'66","reported_seq":201,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.590324+0000","last_change":"2024-04-25T11:45:07.590324+0000","last_active":"2024-04-25T11:45:07.590324+0000","last_peered":"2024-04-25T11:45:07.590324+0000","last_clean":"2024-04-25T11:45:07.590324+0000","last_became_active":"2024-04-25T11:20:19.003670+0000","last_became_peered":"2024-04-25T11:20:19.003670+0000","last_unstale":"2024-04-25T11:45:07.590324+0000","last_undegraded":"2024-04-25T11:45:07.590324+0000","last_fullsized":"2024-04-25T11:45:07.590324+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'66","last_scrub_stamp":"2024-04-25T11:45:07.590202+0000","last_deep_scrub":"40'66","last_deep_scrub_stamp":"2024-04-25T11:45:07.590202+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.590202+0000","objects_scrubbed":4,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:35.941465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90465,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":138,"num_write":57,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"40'5066","reported_seq":5181,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.056629+0000","last_change":"2024-04-25T11:45:18.056774+0000","last_active":"2024-04-25T11:45:18.056629+0000","last_peered":"2024-04-25T11:45:18.056629+0000","last_clean":"2024-04-25T11:45:18.056629+0000","last_became_active":"2024-04-25T11:20:24.068469+0000","last_became_peered":"2024-04-25T11:20:24.068469+0000","last_unstale":"2024-04-25T11:45:18.056629+0000","last_undegraded":"2024-04-25T11:45:18.056629+0000","last_fullsized":"2024-04-25T11:45:18.056629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5066","last_scrub_stamp":"2024-04-25T11:45:18.056251+0000","last_deep_scrub":"40'5066","last_deep_scrub_stamp":"2024-04-25T11:45:18.056251+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.056251+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:01.524201+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":0,"num_write":21672,"num_write_kb":504164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"40'6","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.998276+0000","last_change":"2024-04-25T11:45:13.998441+0000","last_active":"2024-04-25T11:45:13.998276+0000","last_peered":"2024-04-25T11:45:13.998276+0000","last_clean":"2024-04-25T11:45:13.998276+0000","last_became_active":"2024-04-25T11:20:20.022621+0000","last_became_peered":"2024-04-25T11:20:20.022621+0000","last_unstale":"2024-04-25T11:45:13.998276+0000","last_undegraded":"2024-04-25T11:45:13.998276+0000","last_fullsized":"2024-04-25T11:45:13.998276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:13.998024+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:13.998024+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.998024+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:36.008629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'19","reported_seq":98,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.667771+0000","last_change":"2024-04-25T11:45:13.667844+0000","last_active":"2024-04-25T11:45:13.667771+0000","last_peered":"2024-04-25T11:45:13.667771+0000","last_clean":"2024-04-25T11:45:13.667771+0000","last_became_active":"2024-04-25T11:20:19.038223+0000","last_became_peered":"2024-04-25T11:20:19.038223+0000","last_unstale":"2024-04-25T11:45:13.667771+0000","last_undegraded":"2024-04-25T11:45:13.667771+0000","last_fullsized":"2024-04-25T11:45:13.667771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'19","last_scrub_stamp":"2024-04-25T11:45:13.667567+0000","last_deep_scrub":"40'19","last_deep_scrub_stamp":"2024-04-25T11:45:13.667567+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.667567+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:49:10.180138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"40'5144","reported_seq":5249,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.261794+0000","last_change":"2024-04-25T11:45:20.261944+0000","last_active":"2024-04-25T11:45:20.261794+0000","last_peered":"2024-04-25T11:45:20.261794+0000","last_clean":"2024-04-25T11:45:20.261794+0000","last_became_active":"2024-04-25T11:20:24.155141+0000","last_became_peered":"2024-04-25T11:20:24.155141+0000","last_unstale":"2024-04-25T11:45:20.261794+0000","last_undegraded":"2024-04-25T11:45:20.261794+0000","last_fullsized":"2024-04-25T11:45:20.261794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5144","last_scrub_stamp":"2024-04-25T11:45:20.261504+0000","last_deep_scrub":"40'5144","last_deep_scrub_stamp":"2024-04-25T11:45:20.261504+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.261504+0000","objects_scrubbed":0,"log_size":5144,"log_dups_size":0,"ondisk_log_size":5144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:31.005146+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21523,"num_write_kb":507604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"40'5","reported_seq":77,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.715935+0000","last_change":"2024-04-25T11:45:11.715935+0000","last_active":"2024-04-25T11:45:11.715935+0000","last_peered":"2024-04-25T11:45:11.715935+0000","last_clean":"2024-04-25T11:45:11.715935+0000","last_became_active":"2024-04-25T11:20:20.048042+0000","last_became_peered":"2024-04-25T11:20:20.048042+0000","last_unstale":"2024-04-25T11:45:11.715935+0000","last_undegraded":"2024-04-25T11:45:11.715935+0000","last_fullsized":"2024-04-25T11:45:11.715935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:11.715767+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:11.715767+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.715767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:08.950473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"40'14","reported_seq":105,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.225628+0000","last_change":"2024-04-25T11:45:10.225628+0000","last_active":"2024-04-25T11:45:10.225628+0000","last_peered":"2024-04-25T11:45:10.225628+0000","last_clean":"2024-04-25T11:45:10.225628+0000","last_became_active":"2024-04-25T11:20:19.008644+0000","last_became_peered":"2024-04-25T11:20:19.008644+0000","last_unstale":"2024-04-25T11:45:10.225628+0000","last_undegraded":"2024-04-25T11:45:10.225628+0000","last_fullsized":"2024-04-25T11:45:10.225628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:10.225203+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:10.225203+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.225203+0000","objects_scrubbed":5,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:10.273022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":45,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"40'74","reported_seq":173,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.679843+0000","last_change":"2024-04-25T11:45:11.679928+0000","last_active":"2024-04-25T11:45:11.679843+0000","last_peered":"2024-04-25T11:45:11.679843+0000","last_clean":"2024-04-25T11:45:11.679843+0000","last_became_active":"2024-04-25T11:20:08.214462+0000","last_became_peered":"2024-04-25T11:20:08.214462+0000","last_unstale":"2024-04-25T11:45:11.679843+0000","last_undegraded":"2024-04-25T11:45:11.679843+0000","last_fullsized":"2024-04-25T11:45:11.679843+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'74","last_scrub_stamp":"2024-04-25T11:45:11.679639+0000","last_deep_scrub":"40'74","last_deep_scrub_stamp":"2024-04-25T11:45:11.679639+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.679639+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:19.095500+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"40'5040","reported_seq":5145,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.678036+0000","last_change":"2024-04-25T11:45:24.678036+0000","last_active":"2024-04-25T11:45:24.678036+0000","last_peered":"2024-04-25T11:45:24.678036+0000","last_clean":"2024-04-25T11:45:24.678036+0000","last_became_active":"2024-04-25T11:20:24.165723+0000","last_became_peered":"2024-04-25T11:20:24.165723+0000","last_unstale":"2024-04-25T11:45:24.678036+0000","last_undegraded":"2024-04-25T11:45:24.678036+0000","last_fullsized":"2024-04-25T11:45:24.678036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5040","last_scrub_stamp":"2024-04-25T11:45:24.677516+0000","last_deep_scrub":"40'5040","last_deep_scrub_stamp":"2024-04-25T11:45:24.677516+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.677516+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:00:35.421158+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":21819,"num_write_kb":476160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"40'14","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.739326+0000","last_change":"2024-04-25T11:45:22.739455+0000","last_active":"2024-04-25T11:45:22.739326+0000","last_peered":"2024-04-25T11:45:22.739326+0000","last_clean":"2024-04-25T11:45:22.739326+0000","last_became_active":"2024-04-25T11:20:20.047991+0000","last_became_peered":"2024-04-25T11:20:20.047991+0000","last_unstale":"2024-04-25T11:45:22.739326+0000","last_undegraded":"2024-04-25T11:45:22.739326+0000","last_fullsized":"2024-04-25T11:45:22.739326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:22.739061+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:22.739061+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.739061+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:34.960612+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"40'18","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.634632+0000","last_change":"2024-04-25T11:45:10.634731+0000","last_active":"2024-04-25T11:45:10.634632+0000","last_peered":"2024-04-25T11:45:10.634632+0000","last_clean":"2024-04-25T11:45:10.634632+0000","last_became_active":"2024-04-25T11:20:19.036867+0000","last_became_peered":"2024-04-25T11:20:19.036867+0000","last_unstale":"2024-04-25T11:45:10.634632+0000","last_undegraded":"2024-04-25T11:45:10.634632+0000","last_fullsized":"2024-04-25T11:45:10.634632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:10.634413+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:10.634413+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.634413+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:34.877637+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1984,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"40'5278","reported_seq":5385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.724072+0000","last_change":"2024-04-25T11:45:23.724119+0000","last_active":"2024-04-25T11:45:23.724072+0000","last_peered":"2024-04-25T11:45:23.724072+0000","last_clean":"2024-04-25T11:45:23.724072+0000","last_became_active":"2024-04-25T11:20:24.161741+0000","last_became_peered":"2024-04-25T11:20:24.161741+0000","last_unstale":"2024-04-25T11:45:23.724072+0000","last_undegraded":"2024-04-25T11:45:23.724072+0000","last_fullsized":"2024-04-25T11:45:23.724072+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5278","last_scrub_stamp":"2024-04-25T11:45:23.723931+0000","last_deep_scrub":"40'5278","last_deep_scrub_stamp":"2024-04-25T11:45:23.723931+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.723931+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.244119+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":0,"num_write":22099,"num_write_kb":493620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.093007+0000","last_change":"2024-04-25T11:45:19.093141+0000","last_active":"2024-04-25T11:45:19.093007+0000","last_peered":"2024-04-25T11:45:19.093007+0000","last_clean":"2024-04-25T11:45:19.093007+0000","last_became_active":"2024-04-25T11:20:20.041003+0000","last_became_peered":"2024-04-25T11:20:20.041003+0000","last_unstale":"2024-04-25T11:45:19.093007+0000","last_undegraded":"2024-04-25T11:45:19.093007+0000","last_fullsized":"2024-04-25T11:45:19.093007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:19.092669+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:19.092669+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.092669+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:36.520409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"40'15","reported_seq":92,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.997475+0000","last_change":"2024-04-25T11:45:09.997607+0000","last_active":"2024-04-25T11:45:09.997475+0000","last_peered":"2024-04-25T11:45:09.997475+0000","last_clean":"2024-04-25T11:45:09.997475+0000","last_became_active":"2024-04-25T11:20:19.019106+0000","last_became_peered":"2024-04-25T11:20:19.019106+0000","last_unstale":"2024-04-25T11:45:09.997475+0000","last_undegraded":"2024-04-25T11:45:09.997475+0000","last_fullsized":"2024-04-25T11:45:09.997475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.997165+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.997165+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.997165+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:40.884137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"40'4958","reported_seq":5067,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.185146+0000","last_change":"2024-04-25T11:45:23.185177+0000","last_active":"2024-04-25T11:45:23.185146+0000","last_peered":"2024-04-25T11:45:23.185146+0000","last_clean":"2024-04-25T11:45:23.185146+0000","last_became_active":"2024-04-25T11:21:00.477665+0000","last_became_peered":"2024-04-25T11:21:00.477665+0000","last_unstale":"2024-04-25T11:45:23.185146+0000","last_undegraded":"2024-04-25T11:45:23.185146+0000","last_fullsized":"2024-04-25T11:45:23.185146+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4958","last_scrub_stamp":"2024-04-25T11:45:23.185041+0000","last_deep_scrub":"40'4958","last_deep_scrub_stamp":"2024-04-25T11:45:23.185041+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.185041+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:10.890522+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":36,"num_read_kb":0,"num_write":20958,"num_write_kb":497471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,5],"acting":[0,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.870627+0000","last_change":"2024-04-25T11:45:25.870658+0000","last_active":"2024-04-25T11:45:25.870627+0000","last_peered":"2024-04-25T11:45:25.870627+0000","last_clean":"2024-04-25T11:45:25.870627+0000","last_became_active":"2024-04-25T11:20:20.050932+0000","last_became_peered":"2024-04-25T11:20:20.050932+0000","last_unstale":"2024-04-25T11:45:25.870627+0000","last_undegraded":"2024-04-25T11:45:25.870627+0000","last_fullsized":"2024-04-25T11:45:25.870627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:25.870527+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:25.870527+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.870527+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:20.991868+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"40'14","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.662930+0000","last_change":"2024-04-25T11:45:09.662975+0000","last_active":"2024-04-25T11:45:09.662930+0000","last_peered":"2024-04-25T11:45:09.662930+0000","last_clean":"2024-04-25T11:45:09.662930+0000","last_became_active":"2024-04-25T11:20:19.005003+0000","last_became_peered":"2024-04-25T11:20:19.005003+0000","last_unstale":"2024-04-25T11:45:09.662930+0000","last_undegraded":"2024-04-25T11:45:09.662930+0000","last_fullsized":"2024-04-25T11:45:09.662930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:09.662819+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:09.662819+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.662819+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:09.907533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"40'4822","reported_seq":4934,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.354677+0000","last_change":"2024-04-25T11:45:25.354732+0000","last_active":"2024-04-25T11:45:25.354677+0000","last_peered":"2024-04-25T11:45:25.354677+0000","last_clean":"2024-04-25T11:45:25.354677+0000","last_became_active":"2024-04-25T11:20:24.157033+0000","last_became_peered":"2024-04-25T11:20:24.157033+0000","last_unstale":"2024-04-25T11:45:25.354677+0000","last_undegraded":"2024-04-25T11:45:25.354677+0000","last_fullsized":"2024-04-25T11:45:25.354677+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4822","last_scrub_stamp":"2024-04-25T11:45:25.354535+0000","last_deep_scrub":"40'4822","last_deep_scrub_stamp":"2024-04-25T11:45:25.354535+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.354535+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:45:57.712074+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":20365,"num_write_kb":482486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"40'8","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.749429+0000","last_change":"2024-04-25T11:45:23.749511+0000","last_active":"2024-04-25T11:45:23.749429+0000","last_peered":"2024-04-25T11:45:23.749429+0000","last_clean":"2024-04-25T11:45:23.749429+0000","last_became_active":"2024-04-25T11:20:20.048924+0000","last_became_peered":"2024-04-25T11:20:20.048924+0000","last_unstale":"2024-04-25T11:45:23.749429+0000","last_undegraded":"2024-04-25T11:45:23.749429+0000","last_fullsized":"2024-04-25T11:45:23.749429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:23.749274+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:23.749274+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.749274+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:00:23.304151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"40'8","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.304458+0000","last_change":"2024-04-25T11:45:09.304533+0000","last_active":"2024-04-25T11:45:09.304458+0000","last_peered":"2024-04-25T11:45:09.304458+0000","last_clean":"2024-04-25T11:45:09.304458+0000","last_became_active":"2024-04-25T11:20:19.018725+0000","last_became_peered":"2024-04-25T11:20:19.018725+0000","last_unstale":"2024-04-25T11:45:09.304458+0000","last_undegraded":"2024-04-25T11:45:09.304458+0000","last_fullsized":"2024-04-25T11:45:09.304458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:09.304266+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:09.304266+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.304266+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:34.893006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"40'4349","reported_seq":4453,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.384996+0000","last_change":"2024-04-25T11:45:21.385042+0000","last_active":"2024-04-25T11:45:21.384996+0000","last_peered":"2024-04-25T11:45:21.384996+0000","last_clean":"2024-04-25T11:45:21.384996+0000","last_became_active":"2024-04-25T11:20:24.099520+0000","last_became_peered":"2024-04-25T11:20:24.099520+0000","last_unstale":"2024-04-25T11:45:21.384996+0000","last_undegraded":"2024-04-25T11:45:21.384996+0000","last_fullsized":"2024-04-25T11:45:21.384996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4349","last_scrub_stamp":"2024-04-25T11:45:21.384877+0000","last_deep_scrub":"40'4349","last_deep_scrub_stamp":"2024-04-25T11:45:21.384877+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.384877+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:04.852828+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":18465,"num_write_kb":454587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.719257+0000","last_change":"2024-04-25T11:45:20.719329+0000","last_active":"2024-04-25T11:45:20.719257+0000","last_peered":"2024-04-25T11:45:20.719257+0000","last_clean":"2024-04-25T11:45:20.719257+0000","last_became_active":"2024-04-25T11:20:20.041540+0000","last_became_peered":"2024-04-25T11:20:20.041540+0000","last_unstale":"2024-04-25T11:45:20.719257+0000","last_undegraded":"2024-04-25T11:45:20.719257+0000","last_fullsized":"2024-04-25T11:45:20.719257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:20.719027+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:20.719027+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.719027+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:24.521751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"40'39","reported_seq":143,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.635646+0000","last_change":"2024-04-25T11:45:08.635716+0000","last_active":"2024-04-25T11:45:08.635646+0000","last_peered":"2024-04-25T11:45:08.635646+0000","last_clean":"2024-04-25T11:45:08.635646+0000","last_became_active":"2024-04-25T11:20:19.004150+0000","last_became_peered":"2024-04-25T11:20:19.004150+0000","last_unstale":"2024-04-25T11:45:08.635646+0000","last_undegraded":"2024-04-25T11:45:08.635646+0000","last_fullsized":"2024-04-25T11:45:08.635646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'39","last_scrub_stamp":"2024-04-25T11:45:08.635551+0000","last_deep_scrub":"40'39","last_deep_scrub_stamp":"2024-04-25T11:45:08.635551+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.635551+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:47:14.634807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"40'5276","reported_seq":5383,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.278283+0000","last_change":"2024-04-25T11:45:19.278363+0000","last_active":"2024-04-25T11:45:19.278283+0000","last_peered":"2024-04-25T11:45:19.278283+0000","last_clean":"2024-04-25T11:45:19.278283+0000","last_became_active":"2024-04-25T11:20:24.089873+0000","last_became_peered":"2024-04-25T11:20:24.089873+0000","last_unstale":"2024-04-25T11:45:19.278283+0000","last_undegraded":"2024-04-25T11:45:19.278283+0000","last_fullsized":"2024-04-25T11:45:19.278283+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5276","last_scrub_stamp":"2024-04-25T11:45:19.278149+0000","last_deep_scrub":"40'5276","last_deep_scrub_stamp":"2024-04-25T11:45:19.278149+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.278149+0000","objects_scrubbed":0,"log_size":5276,"log_dups_size":0,"ondisk_log_size":5276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:47.798338+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":21946,"num_write_kb":529842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.457209+0000","last_change":"2024-04-25T11:45:15.457365+0000","last_active":"2024-04-25T11:45:15.457209+0000","last_peered":"2024-04-25T11:45:15.457209+0000","last_clean":"2024-04-25T11:45:15.457209+0000","last_became_active":"2024-04-25T11:20:20.002124+0000","last_became_peered":"2024-04-25T11:20:20.002124+0000","last_unstale":"2024-04-25T11:45:15.457209+0000","last_undegraded":"2024-04-25T11:45:15.457209+0000","last_fullsized":"2024-04-25T11:45:15.457209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:15.456978+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:15.456978+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.456978+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:22.872839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"40'18","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.403829+0000","last_change":"2024-04-25T11:45:11.403927+0000","last_active":"2024-04-25T11:45:11.403829+0000","last_peered":"2024-04-25T11:45:11.403829+0000","last_clean":"2024-04-25T11:45:11.403829+0000","last_became_active":"2024-04-25T11:20:19.029303+0000","last_became_peered":"2024-04-25T11:20:19.029303+0000","last_unstale":"2024-04-25T11:45:11.403829+0000","last_undegraded":"2024-04-25T11:45:11.403829+0000","last_fullsized":"2024-04-25T11:45:11.403829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:11.403629+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:11.403629+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.403629+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:53.627083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1402,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"40'5730","reported_seq":5856,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.946836+0000","last_change":"2024-04-25T11:45:23.946901+0000","last_active":"2024-04-25T11:45:23.946836+0000","last_peered":"2024-04-25T11:45:23.946836+0000","last_clean":"2024-04-25T11:45:23.946836+0000","last_became_active":"2024-04-25T11:20:24.155071+0000","last_became_peered":"2024-04-25T11:20:24.155071+0000","last_unstale":"2024-04-25T11:45:23.946836+0000","last_undegraded":"2024-04-25T11:45:23.946836+0000","last_fullsized":"2024-04-25T11:45:23.946836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5730","last_scrub_stamp":"2024-04-25T11:45:23.946688+0000","last_deep_scrub":"40'5730","last_deep_scrub_stamp":"2024-04-25T11:45:23.946688+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.946688+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:52:02.885338+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":0,"num_write":23847,"num_write_kb":545821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.741050+0000","last_change":"2024-04-25T11:45:24.741121+0000","last_active":"2024-04-25T11:45:24.741050+0000","last_peered":"2024-04-25T11:45:24.741050+0000","last_clean":"2024-04-25T11:45:24.741050+0000","last_became_active":"2024-04-25T11:20:20.049160+0000","last_became_peered":"2024-04-25T11:20:20.049160+0000","last_unstale":"2024-04-25T11:45:24.741050+0000","last_undegraded":"2024-04-25T11:45:24.741050+0000","last_fullsized":"2024-04-25T11:45:24.741050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:24.740857+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:24.740857+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.740857+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:00.538047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"40'14","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.173175+0000","last_change":"2024-04-25T11:45:13.173269+0000","last_active":"2024-04-25T11:45:13.173175+0000","last_peered":"2024-04-25T11:45:13.173175+0000","last_clean":"2024-04-25T11:45:13.173175+0000","last_became_active":"2024-04-25T11:20:19.023821+0000","last_became_peered":"2024-04-25T11:20:19.023821+0000","last_unstale":"2024-04-25T11:45:13.173175+0000","last_undegraded":"2024-04-25T11:45:13.173175+0000","last_fullsized":"2024-04-25T11:45:13.173175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:13.172976+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:13.172976+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.172976+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:35.428316+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"40'4272","reported_seq":4389,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.906529+0000","last_change":"2024-04-25T11:45:24.906659+0000","last_active":"2024-04-25T11:45:24.906529+0000","last_peered":"2024-04-25T11:45:24.906529+0000","last_clean":"2024-04-25T11:45:24.906529+0000","last_became_active":"2024-04-25T11:20:24.162101+0000","last_became_peered":"2024-04-25T11:20:24.162101+0000","last_unstale":"2024-04-25T11:45:24.906529+0000","last_undegraded":"2024-04-25T11:45:24.906529+0000","last_fullsized":"2024-04-25T11:45:24.906529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4272","last_scrub_stamp":"2024-04-25T11:45:24.906194+0000","last_deep_scrub":"40'4272","last_deep_scrub_stamp":"2024-04-25T11:45:24.906194+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.906194+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:12.398388+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":0,"num_write":17563,"num_write_kb":454991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.994561+0000","last_change":"2024-04-25T11:45:11.994699+0000","last_active":"2024-04-25T11:45:11.994561+0000","last_peered":"2024-04-25T11:45:11.994561+0000","last_clean":"2024-04-25T11:45:11.994561+0000","last_became_active":"2024-04-25T11:20:20.018957+0000","last_became_peered":"2024-04-25T11:20:20.018957+0000","last_unstale":"2024-04-25T11:45:11.994561+0000","last_undegraded":"2024-04-25T11:45:11.994561+0000","last_fullsized":"2024-04-25T11:45:11.994561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:11.994331+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:11.994331+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.994331+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:12.042150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"40'19","reported_seq":117,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.932211+0000","last_change":"2024-04-25T11:45:07.932248+0000","last_active":"2024-04-25T11:45:07.932211+0000","last_peered":"2024-04-25T11:45:07.932211+0000","last_clean":"2024-04-25T11:45:07.932211+0000","last_became_active":"2024-04-25T11:20:19.022281+0000","last_became_peered":"2024-04-25T11:20:19.022281+0000","last_unstale":"2024-04-25T11:45:07.932211+0000","last_undegraded":"2024-04-25T11:45:07.932211+0000","last_fullsized":"2024-04-25T11:45:07.932211+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'19","last_scrub_stamp":"2024-04-25T11:45:07.932138+0000","last_deep_scrub":"40'19","last_deep_scrub_stamp":"2024-04-25T11:45:07.932138+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.932138+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:41.248478+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":80,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"40'4282","reported_seq":4385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:24.369522+0000","last_change":"2024-04-25T11:45:24.369563+0000","last_active":"2024-04-25T11:45:24.369522+0000","last_peered":"2024-04-25T11:45:24.369522+0000","last_clean":"2024-04-25T11:45:24.369522+0000","last_became_active":"2024-04-25T11:20:24.155726+0000","last_became_peered":"2024-04-25T11:20:24.155726+0000","last_unstale":"2024-04-25T11:45:24.369522+0000","last_undegraded":"2024-04-25T11:45:24.369522+0000","last_fullsized":"2024-04-25T11:45:24.369522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4282","last_scrub_stamp":"2024-04-25T11:45:24.369372+0000","last_deep_scrub":"40'4282","last_deep_scrub_stamp":"2024-04-25T11:45:24.369372+0000","last_clean_scrub_stamp":"2024-04-25T11:45:24.369372+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:27.595641+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":0,"num_write":17949,"num_write_kb":436074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.711671+0000","last_change":"2024-04-25T11:45:21.711805+0000","last_active":"2024-04-25T11:45:21.711671+0000","last_peered":"2024-04-25T11:45:21.711671+0000","last_clean":"2024-04-25T11:45:21.711671+0000","last_became_active":"2024-04-25T11:20:20.043288+0000","last_became_peered":"2024-04-25T11:20:20.043288+0000","last_unstale":"2024-04-25T11:45:21.711671+0000","last_undegraded":"2024-04-25T11:45:21.711671+0000","last_fullsized":"2024-04-25T11:45:21.711671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.711369+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.711369+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.711369+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:39.139108+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"40'353","reported_seq":455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.456243+0000","last_change":"2024-04-25T11:45:10.456343+0000","last_active":"2024-04-25T11:45:10.456243+0000","last_peered":"2024-04-25T11:45:10.456243+0000","last_clean":"2024-04-25T11:45:10.456243+0000","last_became_active":"2024-04-25T11:20:19.026972+0000","last_became_peered":"2024-04-25T11:20:19.026972+0000","last_unstale":"2024-04-25T11:45:10.456243+0000","last_undegraded":"2024-04-25T11:45:10.456243+0000","last_fullsized":"2024-04-25T11:45:10.456243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'353","last_scrub_stamp":"2024-04-25T11:45:10.456029+0000","last_deep_scrub":"40'353","last_deep_scrub_stamp":"2024-04-25T11:45:10.456029+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.456029+0000","objects_scrubbed":6,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:50.007081+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":12354,"num_write":348,"num_write_kb":4272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"40'4220","reported_seq":4327,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.984701+0000","last_change":"2024-04-25T11:45:18.984744+0000","last_active":"2024-04-25T11:45:18.984701+0000","last_peered":"2024-04-25T11:45:18.984701+0000","last_clean":"2024-04-25T11:45:18.984701+0000","last_became_active":"2024-04-25T11:20:24.084824+0000","last_became_peered":"2024-04-25T11:20:24.084824+0000","last_unstale":"2024-04-25T11:45:18.984701+0000","last_undegraded":"2024-04-25T11:45:18.984701+0000","last_fullsized":"2024-04-25T11:45:18.984701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4220","last_scrub_stamp":"2024-04-25T11:45:18.984548+0000","last_deep_scrub":"40'4220","last_deep_scrub_stamp":"2024-04-25T11:45:18.984548+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.984548+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:47.504736+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":0,"num_write":17734,"num_write_kb":427234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.248931+0000","last_change":"2024-04-25T11:45:13.249000+0000","last_active":"2024-04-25T11:45:13.248931+0000","last_peered":"2024-04-25T11:45:13.248931+0000","last_clean":"2024-04-25T11:45:13.248931+0000","last_became_active":"2024-04-25T11:20:20.023334+0000","last_became_peered":"2024-04-25T11:20:20.023334+0000","last_unstale":"2024-04-25T11:45:13.248931+0000","last_undegraded":"2024-04-25T11:45:13.248931+0000","last_fullsized":"2024-04-25T11:45:13.248931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.248736+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.248736+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.248736+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:52.101758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"40'33","reported_seq":117,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.932343+0000","last_change":"2024-04-25T11:45:12.932422+0000","last_active":"2024-04-25T11:45:12.932343+0000","last_peered":"2024-04-25T11:45:12.932343+0000","last_clean":"2024-04-25T11:45:12.932343+0000","last_became_active":"2024-04-25T11:20:19.037486+0000","last_became_peered":"2024-04-25T11:20:19.037486+0000","last_unstale":"2024-04-25T11:45:12.932343+0000","last_undegraded":"2024-04-25T11:45:12.932343+0000","last_fullsized":"2024-04-25T11:45:12.932343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'33","last_scrub_stamp":"2024-04-25T11:45:12.932164+0000","last_deep_scrub":"40'33","last_deep_scrub_stamp":"2024-04-25T11:45:12.932164+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.932164+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:35.447183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":29,"num_write":32,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"40'4473","reported_seq":4571,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.444385+0000","last_change":"2024-04-25T11:45:18.444414+0000","last_active":"2024-04-25T11:45:18.444385+0000","last_peered":"2024-04-25T11:45:18.444385+0000","last_clean":"2024-04-25T11:45:18.444385+0000","last_became_active":"2024-04-25T11:20:24.092568+0000","last_became_peered":"2024-04-25T11:20:24.092568+0000","last_unstale":"2024-04-25T11:45:18.444385+0000","last_undegraded":"2024-04-25T11:45:18.444385+0000","last_fullsized":"2024-04-25T11:45:18.444385+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4473","last_scrub_stamp":"2024-04-25T11:45:18.444279+0000","last_deep_scrub":"40'4473","last_deep_scrub_stamp":"2024-04-25T11:45:18.444279+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.444279+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:15.678984+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":18511,"num_write_kb":434349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"40'14","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.155314+0000","last_change":"2024-04-25T11:45:16.155376+0000","last_active":"2024-04-25T11:45:16.155314+0000","last_peered":"2024-04-25T11:45:16.155314+0000","last_clean":"2024-04-25T11:45:16.155314+0000","last_became_active":"2024-04-25T11:20:20.032641+0000","last_became_peered":"2024-04-25T11:20:20.032641+0000","last_unstale":"2024-04-25T11:45:16.155314+0000","last_undegraded":"2024-04-25T11:45:16.155314+0000","last_fullsized":"2024-04-25T11:45:16.155314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:16.155116+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:16.155116+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.155116+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:02.653658+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"40'9","reported_seq":96,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.175940+0000","last_change":"2024-04-25T11:45:14.176029+0000","last_active":"2024-04-25T11:45:14.175940+0000","last_peered":"2024-04-25T11:45:14.175940+0000","last_clean":"2024-04-25T11:45:14.175940+0000","last_became_active":"2024-04-25T11:20:19.034420+0000","last_became_peered":"2024-04-25T11:20:19.034420+0000","last_unstale":"2024-04-25T11:45:14.175940+0000","last_undegraded":"2024-04-25T11:45:14.175940+0000","last_fullsized":"2024-04-25T11:45:14.175940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:14.175731+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:14.175731+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.175731+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:11.410437+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"40'4342","reported_seq":4454,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:28.839558+0000","last_change":"2024-04-25T11:45:28.839608+0000","last_active":"2024-04-25T11:45:28.839558+0000","last_peered":"2024-04-25T11:45:28.839558+0000","last_clean":"2024-04-25T11:45:28.839558+0000","last_became_active":"2024-04-25T11:21:00.072344+0000","last_became_peered":"2024-04-25T11:21:00.072344+0000","last_unstale":"2024-04-25T11:45:28.839558+0000","last_undegraded":"2024-04-25T11:45:28.839558+0000","last_fullsized":"2024-04-25T11:45:28.839558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4342","last_scrub_stamp":"2024-04-25T11:45:28.839445+0000","last_deep_scrub":"40'4342","last_deep_scrub_stamp":"2024-04-25T11:45:28.839445+0000","last_clean_scrub_stamp":"2024-04-25T11:45:28.839445+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:56.510470+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":0,"num_write":17952,"num_write_kb":448366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6,4],"acting":[3,2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.187114+0000","last_change":"2024-04-25T11:45:15.187264+0000","last_active":"2024-04-25T11:45:15.187114+0000","last_peered":"2024-04-25T11:45:15.187114+0000","last_clean":"2024-04-25T11:45:15.187114+0000","last_became_active":"2024-04-25T11:20:20.032779+0000","last_became_peered":"2024-04-25T11:20:20.032779+0000","last_unstale":"2024-04-25T11:45:15.187114+0000","last_undegraded":"2024-04-25T11:45:15.187114+0000","last_fullsized":"2024-04-25T11:45:15.187114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:15.186774+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:15.186774+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.186774+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:24.739444+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"40'17","reported_seq":101,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.439896+0000","last_change":"2024-04-25T11:45:14.439974+0000","last_active":"2024-04-25T11:45:14.439896+0000","last_peered":"2024-04-25T11:45:14.439896+0000","last_clean":"2024-04-25T11:45:14.439896+0000","last_became_active":"2024-04-25T11:20:19.045576+0000","last_became_peered":"2024-04-25T11:20:19.045576+0000","last_unstale":"2024-04-25T11:45:14.439896+0000","last_undegraded":"2024-04-25T11:45:14.439896+0000","last_fullsized":"2024-04-25T11:45:14.439896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:14.439671+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:14.439671+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.439671+0000","objects_scrubbed":3,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:53.292692+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"40'3797","reported_seq":3908,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:25.852413+0000","last_change":"2024-04-25T11:45:25.852453+0000","last_active":"2024-04-25T11:45:25.852413+0000","last_peered":"2024-04-25T11:45:25.852413+0000","last_clean":"2024-04-25T11:45:25.852413+0000","last_became_active":"2024-04-25T11:20:24.161871+0000","last_became_peered":"2024-04-25T11:20:24.161871+0000","last_unstale":"2024-04-25T11:45:25.852413+0000","last_undegraded":"2024-04-25T11:45:25.852413+0000","last_fullsized":"2024-04-25T11:45:25.852413+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'3797","last_scrub_stamp":"2024-04-25T11:45:25.852274+0000","last_deep_scrub":"40'3797","last_deep_scrub_stamp":"2024-04-25T11:45:25.852274+0000","last_clean_scrub_stamp":"2024-04-25T11:45:25.852274+0000","objects_scrubbed":0,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:18.142336+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":0,"num_write":15973,"num_write_kb":399489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"40'5425","reported_seq":5547,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.211143+0000","last_change":"2024-04-25T11:45:22.211173+0000","last_active":"2024-04-25T11:45:22.211143+0000","last_peered":"2024-04-25T11:45:22.211143+0000","last_clean":"2024-04-25T11:45:22.211143+0000","last_became_active":"2024-04-25T11:20:24.165474+0000","last_became_peered":"2024-04-25T11:20:24.165474+0000","last_unstale":"2024-04-25T11:45:22.211143+0000","last_undegraded":"2024-04-25T11:45:22.211143+0000","last_fullsized":"2024-04-25T11:45:22.211143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5425","last_scrub_stamp":"2024-04-25T11:45:22.211048+0000","last_deep_scrub":"40'5425","last_deep_scrub_stamp":"2024-04-25T11:45:22.211048+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.211048+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:39.029577+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":0,"num_write":22739,"num_write_kb":545420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"40'9","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.718517+0000","last_change":"2024-04-25T11:45:22.718647+0000","last_active":"2024-04-25T11:45:22.718517+0000","last_peered":"2024-04-25T11:45:22.718517+0000","last_clean":"2024-04-25T11:45:22.718517+0000","last_became_active":"2024-04-25T11:20:59.943130+0000","last_became_peered":"2024-04-25T11:20:59.943130+0000","last_unstale":"2024-04-25T11:45:22.718517+0000","last_undegraded":"2024-04-25T11:45:22.718517+0000","last_fullsized":"2024-04-25T11:45:22.718517+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:22.718193+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:22.718193+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.718193+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:23:10.423674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"40'13","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.212794+0000","last_change":"2024-04-25T11:45:12.212940+0000","last_active":"2024-04-25T11:45:12.212794+0000","last_peered":"2024-04-25T11:45:12.212794+0000","last_clean":"2024-04-25T11:45:12.212794+0000","last_became_active":"2024-04-25T11:20:19.008889+0000","last_became_peered":"2024-04-25T11:20:19.008889+0000","last_unstale":"2024-04-25T11:45:12.212794+0000","last_undegraded":"2024-04-25T11:45:12.212794+0000","last_fullsized":"2024-04-25T11:45:12.212794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:12.212479+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:12.212479+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.212479+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:33.514789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"40'5679","reported_seq":5804,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:22.682475+0000","last_change":"2024-04-25T11:45:22.682475+0000","last_active":"2024-04-25T11:45:22.682475+0000","last_peered":"2024-04-25T11:45:22.682475+0000","last_clean":"2024-04-25T11:45:22.682475+0000","last_became_active":"2024-04-25T11:20:24.086309+0000","last_became_peered":"2024-04-25T11:20:24.086309+0000","last_unstale":"2024-04-25T11:45:22.682475+0000","last_undegraded":"2024-04-25T11:45:22.682475+0000","last_fullsized":"2024-04-25T11:45:22.682475+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5679","last_scrub_stamp":"2024-04-25T11:45:22.682156+0000","last_deep_scrub":"40'5679","last_deep_scrub_stamp":"2024-04-25T11:45:22.682156+0000","last_clean_scrub_stamp":"2024-04-25T11:45:22.682156+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:06.150106+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":0,"num_write":24249,"num_write_kb":579958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"40'17","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:08.679314+0000","last_change":"2024-04-25T11:45:08.679340+0000","last_active":"2024-04-25T11:45:08.679314+0000","last_peered":"2024-04-25T11:45:08.679314+0000","last_clean":"2024-04-25T11:45:08.679314+0000","last_became_active":"2024-04-25T11:20:20.036816+0000","last_became_peered":"2024-04-25T11:20:20.036816+0000","last_unstale":"2024-04-25T11:45:08.679314+0000","last_undegraded":"2024-04-25T11:45:08.679314+0000","last_fullsized":"2024-04-25T11:45:08.679314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:08.679230+0000","last_clean_scrub_stamp":"2024-04-25T11:45:08.679230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:24.690578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"40'8","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.760031+0000","last_change":"2024-04-25T11:45:15.760136+0000","last_active":"2024-04-25T11:45:15.760031+0000","last_peered":"2024-04-25T11:45:15.760031+0000","last_clean":"2024-04-25T11:45:15.760031+0000","last_became_active":"2024-04-25T11:20:19.028559+0000","last_became_peered":"2024-04-25T11:20:19.028559+0000","last_unstale":"2024-04-25T11:45:15.760031+0000","last_undegraded":"2024-04-25T11:45:15.760031+0000","last_fullsized":"2024-04-25T11:45:15.760031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:15.759838+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:15.759838+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.759838+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:44.280027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":40,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"40'18","reported_seq":103,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.381983+0000","last_change":"2024-04-25T11:45:12.381983+0000","last_active":"2024-04-25T11:45:12.381983+0000","last_peered":"2024-04-25T11:45:12.381983+0000","last_clean":"2024-04-25T11:45:12.381983+0000","last_became_active":"2024-04-25T11:20:19.029335+0000","last_became_peered":"2024-04-25T11:20:19.029335+0000","last_unstale":"2024-04-25T11:45:12.381983+0000","last_undegraded":"2024-04-25T11:45:12.381983+0000","last_fullsized":"2024-04-25T11:45:12.381983+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:12.381594+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:12.381594+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.381594+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:45.697934+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.054174+0000","last_change":"2024-04-25T11:45:13.054174+0000","last_active":"2024-04-25T11:45:13.054174+0000","last_peered":"2024-04-25T11:45:13.054174+0000","last_clean":"2024-04-25T11:45:13.054174+0000","last_became_active":"2024-04-25T11:20:20.022416+0000","last_became_peered":"2024-04-25T11:20:20.022416+0000","last_unstale":"2024-04-25T11:45:13.054174+0000","last_undegraded":"2024-04-25T11:45:13.054174+0000","last_fullsized":"2024-04-25T11:45:13.054174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:13.053765+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:13.053765+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.053765+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:35.568783+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"40'4277","reported_seq":4385,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:13.786224+0000","last_change":"2024-04-25T11:45:13.786224+0000","last_active":"2024-04-25T11:45:13.786224+0000","last_peered":"2024-04-25T11:45:13.786224+0000","last_clean":"2024-04-25T11:45:13.786224+0000","last_became_active":"2024-04-25T11:20:24.081050+0000","last_became_peered":"2024-04-25T11:20:24.081050+0000","last_unstale":"2024-04-25T11:45:13.786224+0000","last_undegraded":"2024-04-25T11:45:13.786224+0000","last_fullsized":"2024-04-25T11:45:13.786224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4277","last_scrub_stamp":"2024-04-25T11:45:13.786072+0000","last_deep_scrub":"40'4277","last_deep_scrub_stamp":"2024-04-25T11:45:13.786072+0000","last_clean_scrub_stamp":"2024-04-25T11:45:13.786072+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:44.692918+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":0,"num_write":17490,"num_write_kb":434283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"40'642","reported_seq":726,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.820194+0000","last_change":"2024-04-25T11:45:14.820194+0000","last_active":"2024-04-25T11:45:14.820194+0000","last_peered":"2024-04-25T11:45:14.820194+0000","last_clean":"2024-04-25T11:45:14.820194+0000","last_became_active":"2024-04-25T11:20:19.020171+0000","last_became_peered":"2024-04-25T11:20:19.020171+0000","last_unstale":"2024-04-25T11:45:14.820194+0000","last_undegraded":"2024-04-25T11:45:14.820194+0000","last_fullsized":"2024-04-25T11:45:14.820194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'642","last_scrub_stamp":"2024-04-25T11:45:14.819761+0000","last_deep_scrub":"40'642","last_deep_scrub_stamp":"2024-04-25T11:45:14.819761+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.819761+0000","objects_scrubbed":3,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:07:58.287711+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195856,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12297,"num_write":626,"num_write_kb":4245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"40'11","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.259327+0000","last_change":"2024-04-25T11:45:14.259327+0000","last_active":"2024-04-25T11:45:14.259327+0000","last_peered":"2024-04-25T11:45:14.259327+0000","last_clean":"2024-04-25T11:45:14.259327+0000","last_became_active":"2024-04-25T11:20:20.034965+0000","last_became_peered":"2024-04-25T11:20:20.034965+0000","last_unstale":"2024-04-25T11:45:14.259327+0000","last_undegraded":"2024-04-25T11:45:14.259327+0000","last_fullsized":"2024-04-25T11:45:14.259327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:14.259027+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:14.259027+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.259027+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:21.674888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"40'5271","reported_seq":5384,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.018210+0000","last_change":"2024-04-25T11:45:23.018210+0000","last_active":"2024-04-25T11:45:23.018210+0000","last_peered":"2024-04-25T11:45:23.018210+0000","last_clean":"2024-04-25T11:45:23.018210+0000","last_became_active":"2024-04-25T11:20:24.089315+0000","last_became_peered":"2024-04-25T11:20:24.089315+0000","last_unstale":"2024-04-25T11:45:23.018210+0000","last_undegraded":"2024-04-25T11:45:23.018210+0000","last_fullsized":"2024-04-25T11:45:23.018210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5271","last_scrub_stamp":"2024-04-25T11:45:23.018042+0000","last_deep_scrub":"40'5271","last_deep_scrub_stamp":"2024-04-25T11:45:23.018042+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.018042+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:45:55.375581+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":0,"num_write":22092,"num_write_kb":534564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"40'15","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.487205+0000","last_change":"2024-04-25T11:45:16.487205+0000","last_active":"2024-04-25T11:45:16.487205+0000","last_peered":"2024-04-25T11:45:16.487205+0000","last_clean":"2024-04-25T11:45:16.487205+0000","last_became_active":"2024-04-25T11:20:20.005680+0000","last_became_peered":"2024-04-25T11:20:20.005680+0000","last_unstale":"2024-04-25T11:45:16.487205+0000","last_undegraded":"2024-04-25T11:45:16.487205+0000","last_fullsized":"2024-04-25T11:45:16.487205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:16.486833+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:16.486833+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.486833+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:39.001851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.663691+0000","last_change":"2024-04-25T11:45:12.663691+0000","last_active":"2024-04-25T11:45:12.663691+0000","last_peered":"2024-04-25T11:45:12.663691+0000","last_clean":"2024-04-25T11:45:12.663691+0000","last_became_active":"2024-04-25T11:20:20.032280+0000","last_became_peered":"2024-04-25T11:20:20.032280+0000","last_unstale":"2024-04-25T11:45:12.663691+0000","last_undegraded":"2024-04-25T11:45:12.663691+0000","last_fullsized":"2024-04-25T11:45:12.663691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:12.663209+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:12.663209+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.663209+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:52.079695+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"40'18","reported_seq":91,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.107154+0000","last_change":"2024-04-25T11:45:20.107154+0000","last_active":"2024-04-25T11:45:20.107154+0000","last_peered":"2024-04-25T11:45:20.107154+0000","last_clean":"2024-04-25T11:45:20.107154+0000","last_became_active":"2024-04-25T11:20:20.046611+0000","last_became_peered":"2024-04-25T11:20:20.046611+0000","last_unstale":"2024-04-25T11:45:20.107154+0000","last_undegraded":"2024-04-25T11:45:20.107154+0000","last_fullsized":"2024-04-25T11:45:20.107154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'18","last_scrub_stamp":"2024-04-25T11:45:20.106865+0000","last_deep_scrub":"40'18","last_deep_scrub_stamp":"2024-04-25T11:45:20.106865+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.106865+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:32.328416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"40'9","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.079311+0000","last_change":"2024-04-25T11:45:18.079311+0000","last_active":"2024-04-25T11:45:18.079311+0000","last_peered":"2024-04-25T11:45:18.079311+0000","last_clean":"2024-04-25T11:45:18.079311+0000","last_became_active":"2024-04-25T11:20:20.038337+0000","last_became_peered":"2024-04-25T11:20:20.038337+0000","last_unstale":"2024-04-25T11:45:18.079311+0000","last_undegraded":"2024-04-25T11:45:18.079311+0000","last_fullsized":"2024-04-25T11:45:18.079311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'9","last_scrub_stamp":"2024-04-25T11:45:18.078968+0000","last_deep_scrub":"40'9","last_deep_scrub_stamp":"2024-04-25T11:45:18.078968+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.078968+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:01.546919+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"40'10","reported_seq":81,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.676165+0000","last_change":"2024-04-25T11:45:10.676231+0000","last_active":"2024-04-25T11:45:10.676165+0000","last_peered":"2024-04-25T11:45:10.676165+0000","last_clean":"2024-04-25T11:45:10.676165+0000","last_became_active":"2024-04-25T11:20:20.036616+0000","last_became_peered":"2024-04-25T11:20:20.036616+0000","last_unstale":"2024-04-25T11:45:10.676165+0000","last_undegraded":"2024-04-25T11:45:10.676165+0000","last_fullsized":"2024-04-25T11:45:10.676165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:10.675970+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:10.675970+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.675970+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:33.373767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.989208+0000","last_change":"2024-04-25T11:45:16.989253+0000","last_active":"2024-04-25T11:45:16.989208+0000","last_peered":"2024-04-25T11:45:16.989208+0000","last_clean":"2024-04-25T11:45:16.989208+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:16.989208+0000","last_undegraded":"2024-04-25T11:45:16.989208+0000","last_fullsized":"2024-04-25T11:45:16.989208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:16.989058+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:16.989058+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.989058+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:35.765902+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"40'12","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.582718+0000","last_change":"2024-04-25T11:45:19.582853+0000","last_active":"2024-04-25T11:45:19.582718+0000","last_peered":"2024-04-25T11:45:19.582718+0000","last_clean":"2024-04-25T11:45:19.582718+0000","last_became_active":"2024-04-25T11:20:20.018006+0000","last_became_peered":"2024-04-25T11:20:20.018006+0000","last_unstale":"2024-04-25T11:45:19.582718+0000","last_undegraded":"2024-04-25T11:45:19.582718+0000","last_fullsized":"2024-04-25T11:45:19.582718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:19.582364+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:19.582364+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.582364+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:16.817070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.839653+0000","last_change":"2024-04-25T11:45:17.839732+0000","last_active":"2024-04-25T11:45:17.839653+0000","last_peered":"2024-04-25T11:45:17.839653+0000","last_clean":"2024-04-25T11:45:17.839653+0000","last_became_active":"2024-04-25T11:20:20.022921+0000","last_became_peered":"2024-04-25T11:20:20.022921+0000","last_unstale":"2024-04-25T11:45:17.839653+0000","last_undegraded":"2024-04-25T11:45:17.839653+0000","last_fullsized":"2024-04-25T11:45:17.839653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:17.839416+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:17.839416+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.839416+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:54.958479+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"40'12","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.132888+0000","last_change":"2024-04-25T11:45:21.133023+0000","last_active":"2024-04-25T11:45:21.132888+0000","last_peered":"2024-04-25T11:45:21.132888+0000","last_clean":"2024-04-25T11:45:21.132888+0000","last_became_active":"2024-04-25T11:20:20.039807+0000","last_became_peered":"2024-04-25T11:20:20.039807+0000","last_unstale":"2024-04-25T11:45:21.132888+0000","last_undegraded":"2024-04-25T11:45:21.132888+0000","last_fullsized":"2024-04-25T11:45:21.132888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.132544+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.132544+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.132544+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:18.076691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:12.762701+0000","last_change":"2024-04-25T11:45:12.762800+0000","last_active":"2024-04-25T11:45:12.762701+0000","last_peered":"2024-04-25T11:45:12.762701+0000","last_clean":"2024-04-25T11:45:12.762701+0000","last_became_active":"2024-04-25T11:20:20.034422+0000","last_became_peered":"2024-04-25T11:20:20.034422+0000","last_unstale":"2024-04-25T11:45:12.762701+0000","last_undegraded":"2024-04-25T11:45:12.762701+0000","last_fullsized":"2024-04-25T11:45:12.762701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:12.762495+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:12.762495+0000","last_clean_scrub_stamp":"2024-04-25T11:45:12.762495+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:22.315164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.253003+0000","last_change":"2024-04-25T11:45:09.253079+0000","last_active":"2024-04-25T11:45:09.253003+0000","last_peered":"2024-04-25T11:45:09.253003+0000","last_clean":"2024-04-25T11:45:09.253003+0000","last_became_active":"2024-04-25T11:20:20.032087+0000","last_became_peered":"2024-04-25T11:20:20.032087+0000","last_unstale":"2024-04-25T11:45:09.253003+0000","last_undegraded":"2024-04-25T11:45:09.253003+0000","last_fullsized":"2024-04-25T11:45:09.253003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:09.252761+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:09.252761+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.252761+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:48.105783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"40'17","reported_seq":88,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.018421+0000","last_change":"2024-04-25T11:45:11.018505+0000","last_active":"2024-04-25T11:45:11.018421+0000","last_peered":"2024-04-25T11:45:11.018421+0000","last_clean":"2024-04-25T11:45:11.018421+0000","last_became_active":"2024-04-25T11:20:20.017123+0000","last_became_peered":"2024-04-25T11:20:20.017123+0000","last_unstale":"2024-04-25T11:45:11.018421+0000","last_undegraded":"2024-04-25T11:45:11.018421+0000","last_fullsized":"2024-04-25T11:45:11.018421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.018243+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.018243+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.018243+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:58.678307+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"40'11","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.512474+0000","last_change":"2024-04-25T11:45:17.512613+0000","last_active":"2024-04-25T11:45:17.512474+0000","last_peered":"2024-04-25T11:45:17.512474+0000","last_clean":"2024-04-25T11:45:17.512474+0000","last_became_active":"2024-04-25T11:20:20.017480+0000","last_became_peered":"2024-04-25T11:20:20.017480+0000","last_unstale":"2024-04-25T11:45:17.512474+0000","last_undegraded":"2024-04-25T11:45:17.512474+0000","last_fullsized":"2024-04-25T11:45:17.512474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:17.512177+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:17.512177+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.512177+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:39.522782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"40'13","reported_seq":70,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.752332+0000","last_change":"2024-04-25T11:45:21.752385+0000","last_active":"2024-04-25T11:45:21.752332+0000","last_peered":"2024-04-25T11:45:21.752332+0000","last_clean":"2024-04-25T11:45:21.752332+0000","last_became_active":"2024-04-25T11:20:59.941897+0000","last_became_peered":"2024-04-25T11:20:59.941897+0000","last_unstale":"2024-04-25T11:45:21.752332+0000","last_undegraded":"2024-04-25T11:45:21.752332+0000","last_fullsized":"2024-04-25T11:45:21.752332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:21.752173+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:21.752173+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.752173+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:38.570702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"40'17","reported_seq":87,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.605153+0000","last_change":"2024-04-25T11:45:11.605218+0000","last_active":"2024-04-25T11:45:11.605153+0000","last_peered":"2024-04-25T11:45:11.605153+0000","last_clean":"2024-04-25T11:45:11.605153+0000","last_became_active":"2024-04-25T11:20:20.021342+0000","last_became_peered":"2024-04-25T11:20:20.021342+0000","last_unstale":"2024-04-25T11:45:11.605153+0000","last_undegraded":"2024-04-25T11:45:11.605153+0000","last_fullsized":"2024-04-25T11:45:11.605153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'17","last_scrub_stamp":"2024-04-25T11:45:11.604952+0000","last_deep_scrub":"40'17","last_deep_scrub_stamp":"2024-04-25T11:45:11.604952+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.604952+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:50.401347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:17.297638+0000","last_change":"2024-04-25T11:45:17.297676+0000","last_active":"2024-04-25T11:45:17.297638+0000","last_peered":"2024-04-25T11:45:17.297638+0000","last_clean":"2024-04-25T11:45:17.297638+0000","last_became_active":"2024-04-25T11:20:20.040487+0000","last_became_peered":"2024-04-25T11:20:20.040487+0000","last_unstale":"2024-04-25T11:45:17.297638+0000","last_undegraded":"2024-04-25T11:45:17.297638+0000","last_fullsized":"2024-04-25T11:45:17.297638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:17.297510+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:17.297510+0000","last_clean_scrub_stamp":"2024-04-25T11:45:17.297510+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:06.981200+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"40'13","reported_seq":84,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:16.125400+0000","last_change":"2024-04-25T11:45:16.125440+0000","last_active":"2024-04-25T11:45:16.125400+0000","last_peered":"2024-04-25T11:45:16.125400+0000","last_clean":"2024-04-25T11:45:16.125400+0000","last_became_active":"2024-04-25T11:20:20.038878+0000","last_became_peered":"2024-04-25T11:20:20.038878+0000","last_unstale":"2024-04-25T11:45:16.125400+0000","last_undegraded":"2024-04-25T11:45:16.125400+0000","last_fullsized":"2024-04-25T11:45:16.125400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:16.125305+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:16.125305+0000","last_clean_scrub_stamp":"2024-04-25T11:45:16.125305+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:05.094461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"40'14","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:14.980915+0000","last_change":"2024-04-25T11:45:14.980986+0000","last_active":"2024-04-25T11:45:14.980915+0000","last_peered":"2024-04-25T11:45:14.980915+0000","last_clean":"2024-04-25T11:45:14.980915+0000","last_became_active":"2024-04-25T11:20:20.026134+0000","last_became_peered":"2024-04-25T11:20:20.026134+0000","last_unstale":"2024-04-25T11:45:14.980915+0000","last_undegraded":"2024-04-25T11:45:14.980915+0000","last_fullsized":"2024-04-25T11:45:14.980915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'14","last_scrub_stamp":"2024-04-25T11:45:14.980689+0000","last_deep_scrub":"40'14","last_deep_scrub_stamp":"2024-04-25T11:45:14.980689+0000","last_clean_scrub_stamp":"2024-04-25T11:45:14.980689+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:37.236029+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"40'16","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:15.011310+0000","last_change":"2024-04-25T11:45:15.011379+0000","last_active":"2024-04-25T11:45:15.011310+0000","last_peered":"2024-04-25T11:45:15.011310+0000","last_clean":"2024-04-25T11:45:15.011310+0000","last_became_active":"2024-04-25T11:20:20.026170+0000","last_became_peered":"2024-04-25T11:20:20.026170+0000","last_unstale":"2024-04-25T11:45:15.011310+0000","last_undegraded":"2024-04-25T11:45:15.011310+0000","last_fullsized":"2024-04-25T11:45:15.011310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'16","last_scrub_stamp":"2024-04-25T11:45:15.011069+0000","last_deep_scrub":"40'16","last_deep_scrub_stamp":"2024-04-25T11:45:15.011069+0000","last_clean_scrub_stamp":"2024-04-25T11:45:15.011069+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:20:36.701429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"40'15","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:09.785880+0000","last_change":"2024-04-25T11:45:09.785939+0000","last_active":"2024-04-25T11:45:09.785880+0000","last_peered":"2024-04-25T11:45:09.785880+0000","last_clean":"2024-04-25T11:45:09.785880+0000","last_became_active":"2024-04-25T11:20:20.037635+0000","last_became_peered":"2024-04-25T11:20:20.037635+0000","last_unstale":"2024-04-25T11:45:09.785880+0000","last_undegraded":"2024-04-25T11:45:09.785880+0000","last_fullsized":"2024-04-25T11:45:09.785880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:09.785718+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:09.785718+0000","last_clean_scrub_stamp":"2024-04-25T11:45:09.785718+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:49:06.298289+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"40'10","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:11.291058+0000","last_change":"2024-04-25T11:45:11.291123+0000","last_active":"2024-04-25T11:45:11.291058+0000","last_peered":"2024-04-25T11:45:11.291058+0000","last_clean":"2024-04-25T11:45:11.291058+0000","last_became_active":"2024-04-25T11:20:20.007979+0000","last_became_peered":"2024-04-25T11:20:20.007979+0000","last_unstale":"2024-04-25T11:45:11.291058+0000","last_undegraded":"2024-04-25T11:45:11.291058+0000","last_fullsized":"2024-04-25T11:45:11.291058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'10","last_scrub_stamp":"2024-04-25T11:45:11.290861+0000","last_deep_scrub":"40'10","last_deep_scrub_stamp":"2024-04-25T11:45:11.290861+0000","last_clean_scrub_stamp":"2024-04-25T11:45:11.290861+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:12.768770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"40'11","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:20.620213+0000","last_change":"2024-04-25T11:45:20.620254+0000","last_active":"2024-04-25T11:45:20.620213+0000","last_peered":"2024-04-25T11:45:20.620213+0000","last_clean":"2024-04-25T11:45:20.620213+0000","last_became_active":"2024-04-25T11:20:20.031156+0000","last_became_peered":"2024-04-25T11:20:20.031156+0000","last_unstale":"2024-04-25T11:45:20.620213+0000","last_undegraded":"2024-04-25T11:45:20.620213+0000","last_fullsized":"2024-04-25T11:45:20.620213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:20.620052+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:20.620052+0000","last_clean_scrub_stamp":"2024-04-25T11:45:20.620052+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:30.172722+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"40'12","reported_seq":83,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.562635+0000","last_change":"2024-04-25T11:45:21.562708+0000","last_active":"2024-04-25T11:45:21.562635+0000","last_peered":"2024-04-25T11:45:21.562635+0000","last_clean":"2024-04-25T11:45:21.562635+0000","last_became_active":"2024-04-25T11:20:20.040967+0000","last_became_peered":"2024-04-25T11:20:20.040967+0000","last_unstale":"2024-04-25T11:45:21.562635+0000","last_undegraded":"2024-04-25T11:45:21.562635+0000","last_fullsized":"2024-04-25T11:45:21.562635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'12","last_scrub_stamp":"2024-04-25T11:45:21.562409+0000","last_deep_scrub":"40'12","last_deep_scrub_stamp":"2024-04-25T11:45:21.562409+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.562409+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:40.339253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"40'15","reported_seq":86,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:26.924035+0000","last_change":"2024-04-25T11:45:26.924153+0000","last_active":"2024-04-25T11:45:26.924035+0000","last_peered":"2024-04-25T11:45:26.924035+0000","last_clean":"2024-04-25T11:45:26.924035+0000","last_became_active":"2024-04-25T11:20:20.051195+0000","last_became_peered":"2024-04-25T11:20:20.051195+0000","last_unstale":"2024-04-25T11:45:26.924035+0000","last_undegraded":"2024-04-25T11:45:26.924035+0000","last_fullsized":"2024-04-25T11:45:26.924035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'15","last_scrub_stamp":"2024-04-25T11:45:26.923746+0000","last_deep_scrub":"40'15","last_deep_scrub_stamp":"2024-04-25T11:45:26.923746+0000","last_clean_scrub_stamp":"2024-04-25T11:45:26.923746+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:44.560086+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"40'6","reported_seq":76,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:21.843379+0000","last_change":"2024-04-25T11:45:21.843483+0000","last_active":"2024-04-25T11:45:21.843379+0000","last_peered":"2024-04-25T11:45:21.843379+0000","last_clean":"2024-04-25T11:45:21.843379+0000","last_became_active":"2024-04-25T11:20:20.048279+0000","last_became_peered":"2024-04-25T11:20:20.048279+0000","last_unstale":"2024-04-25T11:45:21.843379+0000","last_undegraded":"2024-04-25T11:45:21.843379+0000","last_fullsized":"2024-04-25T11:45:21.843379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:21.843208+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:21.843208+0000","last_clean_scrub_stamp":"2024-04-25T11:45:21.843208+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:25.069478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.701241+0000","last_change":"2024-04-25T11:45:18.701241+0000","last_active":"2024-04-25T11:45:18.701241+0000","last_peered":"2024-04-25T11:45:18.701241+0000","last_clean":"2024-04-25T11:45:18.701241+0000","last_became_active":"2024-04-25T11:20:20.037009+0000","last_became_peered":"2024-04-25T11:20:20.037009+0000","last_unstale":"2024-04-25T11:45:18.701241+0000","last_undegraded":"2024-04-25T11:45:18.701241+0000","last_fullsized":"2024-04-25T11:45:18.701241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:18.700927+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:18.700927+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.700927+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:08:02.168877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"40'8","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:18.080502+0000","last_change":"2024-04-25T11:45:18.080502+0000","last_active":"2024-04-25T11:45:18.080502+0000","last_peered":"2024-04-25T11:45:18.080502+0000","last_clean":"2024-04-25T11:45:18.080502+0000","last_became_active":"2024-04-25T11:20:20.033029+0000","last_became_peered":"2024-04-25T11:20:20.033029+0000","last_unstale":"2024-04-25T11:45:18.080502+0000","last_undegraded":"2024-04-25T11:45:18.080502+0000","last_fullsized":"2024-04-25T11:45:18.080502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'8","last_scrub_stamp":"2024-04-25T11:45:18.080106+0000","last_deep_scrub":"40'8","last_deep_scrub_stamp":"2024-04-25T11:45:18.080106+0000","last_clean_scrub_stamp":"2024-04-25T11:45:18.080106+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:21.882829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"40'11","reported_seq":82,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:19.779099+0000","last_change":"2024-04-25T11:45:19.779099+0000","last_active":"2024-04-25T11:45:19.779099+0000","last_peered":"2024-04-25T11:45:19.779099+0000","last_clean":"2024-04-25T11:45:19.779099+0000","last_became_active":"2024-04-25T11:20:20.037665+0000","last_became_peered":"2024-04-25T11:20:20.037665+0000","last_unstale":"2024-04-25T11:45:19.779099+0000","last_undegraded":"2024-04-25T11:45:19.779099+0000","last_fullsized":"2024-04-25T11:45:19.779099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'11","last_scrub_stamp":"2024-04-25T11:45:19.778858+0000","last_deep_scrub":"40'11","last_deep_scrub_stamp":"2024-04-25T11:45:19.778858+0000","last_clean_scrub_stamp":"2024-04-25T11:45:19.778858+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:48.299046+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"40'5","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:23.201395+0000","last_change":"2024-04-25T11:45:23.201463+0000","last_active":"2024-04-25T11:45:23.201395+0000","last_peered":"2024-04-25T11:45:23.201395+0000","last_clean":"2024-04-25T11:45:23.201395+0000","last_became_active":"2024-04-25T11:20:20.041971+0000","last_became_peered":"2024-04-25T11:20:20.041971+0000","last_unstale":"2024-04-25T11:45:23.201395+0000","last_undegraded":"2024-04-25T11:45:23.201395+0000","last_fullsized":"2024-04-25T11:45:23.201395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'5","last_scrub_stamp":"2024-04-25T11:45:23.201077+0000","last_deep_scrub":"40'5","last_deep_scrub_stamp":"2024-04-25T11:45:23.201077+0000","last_clean_scrub_stamp":"2024-04-25T11:45:23.201077+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:18.322418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"40'13","reported_seq":85,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.025609+0000","last_change":"2024-04-25T11:45:10.025609+0000","last_active":"2024-04-25T11:45:10.025609+0000","last_peered":"2024-04-25T11:45:10.025609+0000","last_clean":"2024-04-25T11:45:10.025609+0000","last_became_active":"2024-04-25T11:20:20.013538+0000","last_became_peered":"2024-04-25T11:20:20.013538+0000","last_unstale":"2024-04-25T11:45:10.025609+0000","last_undegraded":"2024-04-25T11:45:10.025609+0000","last_fullsized":"2024-04-25T11:45:10.025609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'13","last_scrub_stamp":"2024-04-25T11:45:10.025413+0000","last_deep_scrub":"40'13","last_deep_scrub_stamp":"2024-04-25T11:45:10.025413+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.025413+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:40.912385+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"40'6","reported_seq":79,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:07.076436+0000","last_change":"2024-04-25T11:45:07.076436+0000","last_active":"2024-04-25T11:45:07.076436+0000","last_peered":"2024-04-25T11:45:07.076436+0000","last_clean":"2024-04-25T11:45:07.076436+0000","last_became_active":"2024-04-25T11:20:20.029328+0000","last_became_peered":"2024-04-25T11:20:20.029328+0000","last_unstale":"2024-04-25T11:45:07.076436+0000","last_undegraded":"2024-04-25T11:45:07.076436+0000","last_fullsized":"2024-04-25T11:45:07.076436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'6","last_scrub_stamp":"2024-04-25T11:45:07.076217+0000","last_deep_scrub":"40'6","last_deep_scrub_stamp":"2024-04-25T11:45:07.076217+0000","last_clean_scrub_stamp":"2024-04-25T11:45:07.076217+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:49.299671+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"40'7","reported_seq":80,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-25T11:45:10.780571+0000","last_change":"2024-04-25T11:45:10.780571+0000","last_active":"2024-04-25T11:45:10.780571+0000","last_peered":"2024-04-25T11:45:10.780571+0000","last_clean":"2024-04-25T11:45:10.780571+0000","last_became_active":"2024-04-25T11:20:20.037660+0000","last_became_peered":"2024-04-25T11:20:20.037660+0000","last_unstale":"2024-04-25T11:45:10.780571+0000","last_undegraded":"2024-04-25T11:45:10.780571+0000","last_fullsized":"2024-04-25T11:45:10.780571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'7","last_scrub_stamp":"2024-04-25T11:45:10.780432+0000","last_deep_scrub":"40'7","last_deep_scrub_stamp":"2024-04-25T11:45:10.780432+0000","last_clean_scrub_stamp":"2024-04-25T11:45:10.780432+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:49.633454+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1446,"num_read_kb":0,"num_write":654935,"num_write_kb":15718582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156929,"ondisk_log_size":156929,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":15319788,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1072,"num_read_kb":33356,"num_write":3396,"num_write_kb":16253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30883840,"data_stored":30639576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3229,"ondisk_log_size":3229,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575159,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1000184,"kb_used_data":11824,"kb_used_omap":1,"kb_used_meta":988350,"kb_avail":93371656,"statfs":{"total":96636764160,"available":95612575744,"internally_reserved":0,"allocated":12107776,"data_stored":11840562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":1012070859},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1181784,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1177214,"kb_avail":93190056,"statfs":{"total":96636764160,"available":95426617344,"internally_reserved":0,"allocated":4669440,"data_stored":4418265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1205467590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575160,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776304,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":775806,"kb_avail":93595536,"statfs":{"total":96636764160,"available":95841828864,"internally_reserved":0,"allocated":499712,"data_stored":226431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":794425797},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575161,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1131992,"kb_used_data":4560,"kb_used_omap":1,"kb_used_meta":1127422,"kb_avail":93239848,"statfs":{"total":96636764160,"available":95477604352,"internally_reserved":0,"allocated":4669440,"data_stored":4418133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154480581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575157,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465336,"kb_used_data":8752,"kb_used_omap":1,"kb_used_meta":1456574,"kb_avail":92906504,"statfs":{"total":96636764160,"available":95136260096,"internally_reserved":0,"allocated":8962048,"data_stored":8703072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1491532231},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575158,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857148,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":856574,"kb_avail":93514692,"statfs":{"total":96636764160,"available":95759044608,"internally_reserved":0,"allocated":577536,"data_stored":314314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877132229},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575157,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1484956,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":1484478,"kb_avail":92886884,"statfs":{"total":96636764160,"available":95116169216,"internally_reserved":0,"allocated":479232,"data_stored":223217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1520105926},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575158,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054448,"kb_used_data":3624,"kb_used_omap":1,"kb_used_meta":1050814,"kb_avail":93317392,"statfs":{"total":96636764160,"available":95557009408,"internally_reserved":0,"allocated":3710976,"data_stored":3449798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1076033989},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2666496,"data_stored":2637745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":92629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8481387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4196580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11063296,"data_stored":11028509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T11:45:51.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T11:45:51.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 Processor -- start 2024-04-25T11:45:51.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 -- start start 2024-04-25T11:45:51.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc0b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f70bc0cf490 0x7f70bc0cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70bc0ca5a0 con 0x7f70bc15a3c0 2024-04-25T11:45:51.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f70bc159f30 con 0x7f70bc0cf490 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70c1ac1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f70bc0c4880 con 0x7f70bc154080 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70baffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f70bc0cf490 0x7f70bc0cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70baffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f70bc0cf490 0x7f70bc0cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38498/0 (socket says 172.21.15.136:38498) 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.479+0000 7f70baffd640 1 -- 172.21.15.136:0/942337186 learned_addr learned my addr 172.21.15.136:0/942337186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70bb7fe640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc0b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70baffd640 1 -- 172.21.15.136:0/942337186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 msgr2=0x7f70bc0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70baffd640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc0b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70baffd640 1 -- 172.21.15.136:0/942337186 >> v1:172.21.15.157:6789/0 conn(0x7f70bc15a3c0 legacy=0x7f70bc15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70baffd640 1 -- 172.21.15.136:0/942337186 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70bc0ca840 con 0x7f70bc0cf490 2024-04-25T11:45:51.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70bb7fe640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc0b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:51.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70baffd640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc0cf490 0x7f70bc0cf8a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f70bc15a0b0 tx=0x7f70ac01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3161a3253ced31c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70ba7fc640 1 -- 172.21.15.136:0/942337186 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0b0070 con 0x7f70bc0cf490 2024-04-25T11:45:51.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70ba7fc640 1 -- 172.21.15.136:0/942337186 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ac0ab600 con 0x7f70bc0cf490 2024-04-25T11:45:51.469 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.483+0000 7f70ba7fc640 1 -- 172.21.15.136:0/942337186 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0ab9d0 con 0x7f70bc0cf490 2024-04-25T11:45:51.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- 172.21.15.136:0/942337186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc0cf490 msgr2=0x7f70bc0cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc0cf490 0x7f70bc0cf8a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f70bc15a0b0 tx=0x7f70ac01c500 comp rx=0 tx=0).stop 2024-04-25T11:45:51.470 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- 172.21.15.136:0/942337186 shutdown_connections 2024-04-25T11:45:51.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc0cf490 0x7f70bc0cf8a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/942337186 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc0b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- 172.21.15.136:0/942337186 >> 172.21.15.136:0/942337186 conn(0x7f70bc070f70 msgr2=0x7f70bc068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:51.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- 172.21.15.136:0/942337186 shutdown_connections 2024-04-25T11:45:51.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- 172.21.15.136:0/942337186 wait complete. 2024-04-25T11:45:51.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 Processor -- start 2024-04-25T11:45:51.472 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- start start 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70bc0cf490 0x7f70bc150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc163600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f70bc0cab20 con 0x7f70bc15a3c0 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f70bc154490 con 0x7f70bc0cf490 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70c1ac1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f70bc0c4520 con 0x7f70bc154080 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38510/0 (socket says 172.21.15.136:38510) 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 -- 172.21.15.136:0/1572457142 learned_addr learned my addr 172.21.15.136:0/1572457142 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bb7fe640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70bc0cf490 0x7f70bc150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70baffd640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc163600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 -- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 msgr2=0x7f70bc163600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc163600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 -- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70bc0cf490 msgr2=0x7f70bc150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70bc0cf490 0x7f70bc150780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.487+0000 7f70bbfff640 1 -- 172.21.15.136:0/1572457142 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70bc163b40 con 0x7f70bc15a3c0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70bbfff640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f70b400b9a0 tx=0x7f70b400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4019070 con 0x7f70bc15a3c0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400cba0 con 0x7f70bc15a3c0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70bc150530 con 0x7f70bc15a3c0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4014440 con 0x7f70bc15a3c0 2024-04-25T11:45:51.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70bc162b20 con 0x7f70bc15a3c0 2024-04-25T11:45:51.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70bc06a350 con 0x7f70bc15a3c0 2024-04-25T11:45:51.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f70b401c070 con 0x7f70bc15a3c0 2024-04-25T11:45:51.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 0x7f70a8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f70b408f800 con 0x7f70bc15a3c0 2024-04-25T11:45:51.477 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70bb7fe640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 0x7f70a8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.478 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.491+0000 7f70bb7fe640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 0x7f70a8071910 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f70b0006fd0 tx=0x7f70b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.481 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.495+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f70b405be30 con 0x7f70bc15a3c0 2024-04-25T11:45:51.644 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.659+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f70bc0cd820 con 0x7f70bc15a3c0 2024-04-25T11:45:51.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f70b405b730 con 0x7f70bc15a3c0 2024-04-25T11:45:51.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70a3fff640 1 -- 172.21.15.136:0/1572457142 <== mon.0 v2:172.21.15.136:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f70b405b240 con 0x7f70bc15a3c0 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 msgr2=0x7f70a8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 0x7f70a8071910 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f70b0006fd0 tx=0x7f70b0008040 comp rx=0 tx=0).stop 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 msgr2=0x7f70bc14f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f70b400b9a0 tx=0x7f70b400be60 comp rx=0 tx=0).stop 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 shutdown_connections 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70bc15a3c0 0x7f70bc14f730 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f70a806f400 0x7f70a8071910 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.653 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f70bc154080 0x7f70bc163600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 --2- 172.21.15.136:0/1572457142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70bc0cf490 0x7f70bc150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 >> 172.21.15.136:0/1572457142 conn(0x7f70bc070f70 msgr2=0x7f70bc0cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:51.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 shutdown_connections 2024-04-25T11:45:51.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.667+0000 7f70c1ac1640 1 -- 172.21.15.136:0/1572457142 wait complete. 2024-04-25T11:45:51.671 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T11:45:51.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:45:51.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.827+0000 7f0492628640 1 Processor -- start 2024-04-25T11:45:51.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.827+0000 7f0492628640 1 -- start start 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0492628640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f048c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0492628640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048c0c9af0 con 0x7f048c0c79e0 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0492628640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048c0cb220 con 0x7f048c0c96e0 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0492628640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f048c06cc10 con 0x7f048c15aae0 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f048c0c79e0 0x7f048c15a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:42124/0 (socket says 172.21.15.136:42124) 2024-04-25T11:45:51.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048b7fe640 1 -- 172.21.15.136:0/966643516 learned_addr learned my addr 172.21.15.136:0/966643516 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f048c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1926344547 0 0) 0x7f048c0c9af0 con 0x7f048c0c79e0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0474003530 con 0x7f048c0c79e0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3650978538 0 0) 0x7f0474003530 con 0x7f048c0c79e0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 msgr2=0x7f048c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f048c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 >> v1:172.21.15.157:6789/0 conn(0x7f048c0c96e0 legacy=0x7f048c1540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c0cb500 con 0x7f048c0c79e0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f04800016a0 con 0x7f048c0c79e0 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 >> v1:172.21.15.136:6789/0 conn(0x7f048c0c79e0 legacy=0x7f048c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04740039f0 0x7f0474003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f0474004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.818 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0474004d50 0x7f0474005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f048c06cc10 con 0x7f0474004d50 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f047400bb00 con 0x7f04740039f0 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f047400bd00 con 0x7f048c15aae0 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0474004d50 0x7f0474005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048bfff640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04740039f0 0x7f0474003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f048b7fe640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f0474004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 -- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 msgr2=0x7f0474004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c15aae0 0x7f0474004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 -- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04740039f0 msgr2=0x7f0474003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04740039f0 0x7f0474003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.819 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.831+0000 7f0490bc5640 1 -- 172.21.15.136:0/966643516 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0474003530 con 0x7f0474004d50 2024-04-25T11:45:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0490bc5640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0474004d50 0x7f0474005160 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0484009f90 tx=0x7f048401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48267cfe8d17adac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04840b2070 con 0x7f0474004d50 2024-04-25T11:45:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f04840a5c70 con 0x7f0474004d50 2024-04-25T11:45:51.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f048affd640 1 -- 172.21.15.136:0/966643516 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484009c10 con 0x7f0474004d50 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 -- 172.21.15.136:0/966643516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0474004d50 msgr2=0x7f0474005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 --2- 172.21.15.136:0/966643516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0474004d50 0x7f0474005160 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0484009f90 tx=0x7f048401cc60 comp rx=0 tx=0).stop 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0490bc5640 1 -- 172.21.15.136:0/966643516 reap_dead start 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 -- 172.21.15.136:0/966643516 shutdown_connections 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 -- 172.21.15.136:0/966643516 >> 172.21.15.136:0/966643516 conn(0x7f048c06f9d0 msgr2=0x7f048c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 -- 172.21.15.136:0/966643516 shutdown_connections 2024-04-25T11:45:51.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.835+0000 7f0492628640 1 -- 172.21.15.136:0/966643516 wait complete. 2024-04-25T11:45:51.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 Processor -- start 2024-04-25T11:45:51.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- start start 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 0x7f048c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c154a40 0x7f048c1442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f048c0ced40 con 0x7f048c1f0660 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f048c0cf0f0 con 0x7f048c1f4750 2024-04-25T11:45:51.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f048c1f42c0 con 0x7f048c154a40 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38520/0 (socket says 172.21.15.136:38520) 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 -- 172.21.15.136:0/1667403477 learned_addr learned my addr 172.21.15.136:0/1667403477 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048bfff640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 0x7f048c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0490bc5640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c154a40 0x7f048c1442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 -- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c154a40 msgr2=0x7f048c1442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c154a40 0x7f048c1442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 -- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 msgr2=0x7f048c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 0x7f048c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 -- 172.21.15.136:0/1667403477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c158900 con 0x7f048c1f0660 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048bfff640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 0x7f048c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f048b7fe640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f048000e780 tx=0x7f048000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048000c6c0 con 0x7f048c1f0660 2024-04-25T11:45:51.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0480018070 con 0x7f048c1f0660 2024-04-25T11:45:51.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048c158be0 con 0x7f048c1f0660 2024-04-25T11:45:51.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048000cac0 con 0x7f048c1f0660 2024-04-25T11:45:51.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048c1f3f50 con 0x7f048c1f0660 2024-04-25T11:45:51.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.839+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f048c06a350 con 0x7f048c1f0660 2024-04-25T11:45:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.843+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f048000cc60 con 0x7f048c1f0660 2024-04-25T11:45:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.843+0000 7f0488ff9640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 0x7f0478071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.843+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f048008f9d0 con 0x7f048c1f0660 2024-04-25T11:45:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.843+0000 7f048bfff640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 0x7f0478071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:51.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.843+0000 7f048bfff640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 0x7f0478071910 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f048c153dc0 tx=0x7f047c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:51.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:51.847+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f048005c090 con 0x7f048c1f0660 2024-04-25T11:45:52.020 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.035+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f048c1579a0 con 0x7f048c1f0660 2024-04-25T11:45:52.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.035+0000 7f0488ff9640 1 -- 172.21.15.136:0/1667403477 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+6937 (secure 0 0 0) 0x7f0480064080 con 0x7f048c1f0660 2024-04-25T11:45:52.021 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:52.021 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6852,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2988117878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2988117878},{"type":"v1","addr":"172.21.15.136:6837","nonce":2988117878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:41.473400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5663]},"id":1}]} 2024-04-25T11:45:52.022 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 103 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 msgr2=0x7f0478071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 0x7f0478071910 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f048c153dc0 tx=0x7f047c009040 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 msgr2=0x7f048c156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f048000e780 tx=0x7f048000ec40 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 shutdown_connections 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f047806f400 0x7f0478071910 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f048c154a40 0x7f048c1442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f048c1f0660 0x7f048c156ca0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 --2- 172.21.15.136:0/1667403477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f048c1f4750 0x7f048c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 >> 172.21.15.136:0/1667403477 conn(0x7f048c06f9d0 msgr2=0x7f048c149e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 shutdown_connections 2024-04-25T11:45:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.039+0000 7f0492628640 1 -- 172.21.15.136:0/1667403477 wait complete. 2024-04-25T11:45:52.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T11:45:52.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 Processor -- start 2024-04-25T11:45:52.191 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 -- start start 2024-04-25T11:45:52.192 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c15a5f0 0x7fd20c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd20c15aa00 0x7fd20c0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd20c0c3880 con 0x7fd20c154060 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd20c0c8fd0 con 0x7fd20c15aa00 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd2121b4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c06d9b0 con 0x7fd20c15a5f0 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd20c154060 0x7fd20c0c8800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:42138/0 (socket says 172.21.15.136:42138) 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20b7fe640 1 -- 172.21.15.136:0/2264540639 learned_addr learned my addr 172.21.15.136:0/2264540639 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527474383 0 0) 0x7fd20c0c3880 con 0x7fd20c154060 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20affd640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c15a5f0 0x7fd20c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 --2- 172.21.15.136:0/2264540639 >> v2:172.21.15.157:3300/0 conn(0x7fd20c15aa00 0x7fd20c0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.193 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1f4003530 con 0x7fd20c154060 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 377812852 0 0) 0x7fd1f4003530 con 0x7fd20c154060 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c15a5f0 msgr2=0x7fd20c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c15a5f0 0x7fd20c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 >> v2:172.21.15.157:3300/0 conn(0x7fd20c15aa00 msgr2=0x7fd20c0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 --2- 172.21.15.136:0/2264540639 >> v2:172.21.15.157:3300/0 conn(0x7fd20c15aa00 0x7fd20c0c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c154470 con 0x7fd20c154060 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7fd1fc0016a0 con 0x7fd20c154060 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 >> v1:172.21.15.136:6789/0 conn(0x7fd20c154060 legacy=0x7fd20c0c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1f40039c0 0x7fd1f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1f4004810 0x7fd1f4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1f4005160 0x7fd1f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c0c8fd0 con 0x7fd1f4005160 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c06d9b0 con 0x7fd1f40039c0 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f400bf90 con 0x7fd1f4004810 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1f4005160 0x7fd1f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20affd640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1f4004810 0x7fd1f4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20b7fe640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1f40039c0 0x7fd1f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 -- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1f4004810 msgr2=0x7fd1f4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1f4004810 0x7fd1f4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 -- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1f40039c0 msgr2=0x7fd1f4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1f40039c0 0x7fd1f4003e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.207+0000 7fd20bfff640 1 -- 172.21.15.136:0/2264540639 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f4003530 con 0x7fd1f4005160 2024-04-25T11:45:52.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd20bfff640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1f4005160 0x7fd1f400b9d0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd204009f90 tx=0x7fd20401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ec9302d99fe23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2040b2070 con 0x7fd1f4005160 2024-04-25T11:45:52.197 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd2040a5c90 con 0x7fd1f4005160 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd20a7fc640 1 -- 172.21.15.136:0/2264540639 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd204009c10 con 0x7fd1f4005160 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 -- 172.21.15.136:0/2264540639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1f4005160 msgr2=0x7fd1f400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 --2- 172.21.15.136:0/2264540639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1f4005160 0x7fd1f400b9d0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd204009f90 tx=0x7fd20401cc80 comp rx=0 tx=0).stop 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd20bfff640 1 -- 172.21.15.136:0/2264540639 reap_dead start 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 -- 172.21.15.136:0/2264540639 shutdown_connections 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 -- 172.21.15.136:0/2264540639 >> 172.21.15.136:0/2264540639 conn(0x7fd20c0ca5a0 msgr2=0x7fd20c0ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 -- 172.21.15.136:0/2264540639 shutdown_connections 2024-04-25T11:45:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.211+0000 7fd2121b4640 1 -- 172.21.15.136:0/2264540639 wait complete. 2024-04-25T11:45:52.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 Processor -- start 2024-04-25T11:45:52.199 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- start start 2024-04-25T11:45:52.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 0x7fd20c0c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.201 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd20c0c5cc0 0x7fd20c14f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c15a0d0 con 0x7fd20c0c6360 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c06df30 con 0x7fd20c0c5cc0 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c159d80 con 0x7fd20c074c90 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38540/0 (socket says 172.21.15.136:38540) 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 -- 172.21.15.136:0/1198710190 learned_addr learned my addr 172.21.15.136:0/1198710190 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20affd640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 0x7fd20c0c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20bfff640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd20c0c5cc0 0x7fd20c14f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 -- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 msgr2=0x7fd20c0c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 0x7fd20c0c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 -- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd20c0c5cc0 msgr2=0x7fd20c14f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd20c0c5cc0 0x7fd20c14f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 -- 172.21.15.136:0/1198710190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c071820 con 0x7fd20c0c6360 2024-04-25T11:45:52.202 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20affd640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 0x7fd20c0c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd20b7fe640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00d820 tx=0x7fd1fc00dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc019070 con 0x7fd20c0c6360 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd1fc0074e0 con 0x7fd20c0c6360 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c0c6c10 con 0x7fd20c0c6360 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc004a30 con 0x7fd20c0c6360 2024-04-25T11:45:52.203 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c0c4fa0 con 0x7fd20c0c6360 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.215+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c06a350 con 0x7fd20c0c6360 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.219+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd1fc01c070 con 0x7fd20c0c6360 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.219+0000 7fd1f3fff640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 0x7fd1f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.219+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7fd1fc08fae0 con 0x7fd20c0c6360 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.219+0000 7fd20affd640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 0x7fd1f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.204 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.219+0000 7fd20affd640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 0x7fd1f8071910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd20c073190 tx=0x7fd200008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.207 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.223+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd1fc023020 con 0x7fd20c0c6360 2024-04-25T11:45:52.406 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd20c0c85e0 con 0x7fd20c0c6360 2024-04-25T11:45:52.407 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd1f3fff640 1 -- 172.21.15.136:0/1198710190 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+6937 (secure 0 0 0) 0x7fd1fc05c280 con 0x7fd20c0c6360 2024-04-25T11:45:52.408 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:52.408 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6852,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2988117878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2988117878},{"type":"v1","addr":"172.21.15.136:6837","nonce":2988117878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T11:20:19.386747+0000","modified":"2024-04-25T11:44:41.473400+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5911,"mds_1":5977,"mds_2":5663,"mds_3":5839,"mds_4":6246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5663":{"gid":5663,"name":"f","rank":2,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.157:6839/3709442027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":3709442027},{"type":"v1","addr":"172.21.15.157:6839","nonce":3709442027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5839":{"gid":5839,"name":"d","rank":3,"incarnation":80,"state":"up:active","state_seq":19,"addr":"172.21.15.157:6836/3569413002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":3569413002},{"type":"v1","addr":"172.21.15.157:6836","nonce":3569413002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5911":{"gid":5911,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6833/2091616278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":2091616278},{"type":"v1","addr":"172.21.15.136:6833","nonce":2091616278}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5977":{"gid":5977,"name":"b","rank":1,"incarnation":92,"state":"up:active","state_seq":16,"addr":"172.21.15.157:6837/3388366693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6835","nonce":3388366693},{"type":"v1","addr":"172.21.15.157:6837","nonce":3388366693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6246":{"gid":6246,"name":"c","rank":4,"incarnation":68,"state":"up:active","state_seq":18,"addr":"172.21.15.136:6835/4278341345","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":4278341345},{"type":"v1","addr":"172.21.15.136:6835","nonce":4278341345}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[6246,5977,5911,5839,5663]},"id":1}]} 2024-04-25T11:45:52.408 INFO:teuthology.orchestra.run.smithi136.stderr:dumped fsmap epoch 103 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 msgr2=0x7fd1f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 0x7fd1f8071910 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd20c073190 tx=0x7fd200008040 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 msgr2=0x7fd20c070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00d820 tx=0x7fd1fc00dce0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 shutdown_connections 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd20c0c5cc0 0x7fd20c14f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7fd1f806f400 0x7fd1f8071910 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd20c074c90 0x7fd20c0c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 --2- 172.21.15.136:0/1198710190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd20c0c6360 0x7fd20c070b90 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.410 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 >> 172.21.15.136:0/1198710190 conn(0x7fd20c0ca5a0 msgr2=0x7fd20c157bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 shutdown_connections 2024-04-25T11:45:52.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.423+0000 7fd2121b4640 1 -- 172.21.15.136:0/1198710190 wait complete. 2024-04-25T11:45:52.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T11:45:52.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 Processor -- start 2024-04-25T11:45:52.567 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 -- start start 2024-04-25T11:45:52.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c8815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8c880c4340 0x7f8c880c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c880c3210 con 0x7f8c8815c300 2024-04-25T11:45:52.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8c8815cae0 con 0x7f8c880c4340 2024-04-25T11:45:52.570 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8fb8f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c880c9d20 con 0x7f8c880c2e00 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8d12a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8c8815c300 0x7f8c880749a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:42148/0 (socket says 172.21.15.136:42148) 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8d12a640 1 -- 172.21.15.136:0/2482654634 learned_addr learned my addr 172.21.15.136:0/2482654634 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c8815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2482654634 >> v2:172.21.15.157:3300/0 conn(0x7f8c880c4340 0x7f8c880c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207402995 0 0) 0x7f8c880c3210 con 0x7f8c8815c300 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c74003530 con 0x7f8c8815c300 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3204709975 0 0) 0x7f8c74003530 con 0x7f8c8815c300 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 msgr2=0x7f8c8815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c8815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 >> v2:172.21.15.157:3300/0 conn(0x7f8c880c4340 msgr2=0x7f8c880c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 --2- 172.21.15.136:0/2482654634 >> v2:172.21.15.157:3300/0 conn(0x7f8c880c4340 0x7f8c880c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c880c4c90 con 0x7f8c8815c300 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f8c7c0016a0 con 0x7f8c8815c300 2024-04-25T11:45:52.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 >> v1:172.21.15.136:6789/0 conn(0x7f8c8815c300 legacy=0x7f8c880749a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c740039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74003f40 0x7f8c74004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c74004d50 0x7f8c74005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8815cae0 con 0x7f8c74004d50 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c880c9d20 con 0x7f8c74003f40 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7400bb60 con 0x7f8c880c2e00 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c74004d50 0x7f8c74005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8d12a640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74003f40 0x7f8c74004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c740039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 -- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 msgr2=0x7f8c740039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c880c2e00 0x7f8c740039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 -- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74003f40 msgr2=0x7f8c74004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74003f40 0x7f8c74004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.583+0000 7f8c8e12c640 1 -- 172.21.15.136:0/2482654634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c74003530 con 0x7f8c74004d50 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c74004d50 0x7f8c74005160 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000f240 tx=0x7f8c8001dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa7e8a44097e7a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000ad20 con 0x7f8c74004d50 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f8c800a68e0 con 0x7f8c74004d50 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8c929640 1 -- 172.21.15.136:0/2482654634 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c800a6cf0 con 0x7f8c74004d50 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2482654634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c74004d50 msgr2=0x7f8c74005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2482654634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c74004d50 0x7f8c74005160 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000f240 tx=0x7f8c8001dd80 comp rx=0 tx=0).stop 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8e12c640 1 -- 172.21.15.136:0/2482654634 reap_dead start 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2482654634 shutdown_connections 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2482654634 >> 172.21.15.136:0/2482654634 conn(0x7f8c880ca5a0 msgr2=0x7f8c880ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2482654634 shutdown_connections 2024-04-25T11:45:52.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.587+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2482654634 wait complete. 2024-04-25T11:45:52.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 Processor -- start 2024-04-25T11:45:52.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- start start 2024-04-25T11:45:52.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 0x7f8c881f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c881e5eb0 0x7f8c881e62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c880750b0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8c880ca000 con 0x7f8c74004d50 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8c88073200 con 0x7f8c881e5eb0 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38556/0 (socket says 172.21.15.136:38556) 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 -- 172.21.15.136:0/2386609506 learned_addr learned my addr 172.21.15.136:0/2386609506 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 0x7f8c881f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8e12c640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c881e5eb0 0x7f8c881e62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 -- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c881e5eb0 msgr2=0x7f8c881e62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c881e5eb0 0x7f8c881e62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 -- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 msgr2=0x7f8c881f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.578 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 0x7f8c881f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 -- 172.21.15.136:0/2386609506 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c881f3d10 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 0x7f8c881f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8d12a640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000fe0 tx=0x7f8c7c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c019070 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f8c7c00de50 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c881e7a60 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0047f0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c881f77f0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.591+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c8806a350 con 0x7f8c8814b4d0 2024-04-25T11:45:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.595+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c7c01c070 con 0x7f8c8814b4d0 2024-04-25T11:45:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.595+0000 7f8c7a7fc640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 0x7f8c6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.595+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f8c7c08f6c0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.595+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 0x7f8c6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.580 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.595+0000 7f8c8d92b640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 0x7f8c6c071910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009a00 tx=0x7f8c84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.583 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.599+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8c7c05bcd0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.739 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.755+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c8806d410 con 0x7f8c8814b4d0 2024-04-25T11:45:52.741 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.755+0000 7f8c7a7fc640 1 -- 172.21.15.136:0/2386609506 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+20778 (secure 0 0 0) 0x7f8c7c05b5d0 con 0x7f8c8814b4d0 2024-04-25T11:45:52.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-25T11:45:52.741 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"fsid":"6dead98d-43fa-4435-a3fe-cc7c15301a50","created":"2024-04-25T11:19:54.047126+0000","modified":"2024-04-25T11:44:19.077280+0000","last_up_change":"2024-04-25T11:20:05.489850+0000","last_in_change":"2024-04-25T11:19:56.659894+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T11:20:06.800284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T11:20:17.084245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T11:20:18.360664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T11:20:22.053112+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23f55b63-4108-4987-b1b4-0059929a9aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6801","nonce":3420887400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6803","nonce":3420887400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6807","nonce":3420887400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3420887400},{"type":"v1","addr":"172.21.15.136:6805","nonce":3420887400}]},"public_addr":"172.21.15.136:6801/3420887400","cluster_addr":"172.21.15.136:6803/3420887400","heartbeat_back_addr":"172.21.15.136:6807/3420887400","heartbeat_front_addr":"172.21.15.136:6805/3420887400","state":["exists","up"]},{"osd":1,"uuid":"4b178c77-9ab9-43ea-842b-66df2de40110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6825","nonce":1933356212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6827","nonce":1933356212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6831","nonce":1933356212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1933356212},{"type":"v1","addr":"172.21.15.136:6829","nonce":1933356212}]},"public_addr":"172.21.15.136:6825/1933356212","cluster_addr":"172.21.15.136:6827/1933356212","heartbeat_back_addr":"172.21.15.136:6831/1933356212","heartbeat_front_addr":"172.21.15.136:6829/1933356212","state":["exists","up"]},{"osd":2,"uuid":"8eb9fd06-e438-4799-a0d9-3a3ed2faa796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6817","nonce":4197312530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4197312530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4197312530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":4197312530},{"type":"v1","addr":"172.21.15.136:6821","nonce":4197312530}]},"public_addr":"172.21.15.136:6817/4197312530","cluster_addr":"172.21.15.136:6819/4197312530","heartbeat_back_addr":"172.21.15.136:6823/4197312530","heartbeat_front_addr":"172.21.15.136:6821/4197312530","state":["exists","up"]},{"osd":3,"uuid":"63bae1d0-386d-4a9f-9f2e-9db121da837d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6809","nonce":2427382821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6811","nonce":2427382821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6815","nonce":2427382821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2427382821},{"type":"v1","addr":"172.21.15.136:6813","nonce":2427382821}]},"public_addr":"172.21.15.136:6809/2427382821","cluster_addr":"172.21.15.136:6811/2427382821","heartbeat_back_addr":"172.21.15.136:6815/2427382821","heartbeat_front_addr":"172.21.15.136:6813/2427382821","state":["exists","up"]},{"osd":4,"uuid":"c71fefe5-e003-4765-b8ac-accf933992ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6810","nonce":3530978019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6814","nonce":3530978019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6822","nonce":3530978019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":3530978019},{"type":"v1","addr":"172.21.15.157:6818","nonce":3530978019}]},"public_addr":"172.21.15.157:6810/3530978019","cluster_addr":"172.21.15.157:6814/3530978019","heartbeat_back_addr":"172.21.15.157:6822/3530978019","heartbeat_front_addr":"172.21.15.157:6818/3530978019","state":["exists","up"]},{"osd":5,"uuid":"3a3aa176-27ff-4960-b063-b0b3c6c4daa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215140907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215140907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215140907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215140907},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215140907}]},"public_addr":"172.21.15.157:6801/4215140907","cluster_addr":"172.21.15.157:6803/4215140907","heartbeat_back_addr":"172.21.15.157:6807/4215140907","heartbeat_front_addr":"172.21.15.157:6805/4215140907","state":["exists","up"]},{"osd":6,"uuid":"4eb1c4e4-121a-4669-b755-7f86a53e2e23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6825","nonce":224466234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6827","nonce":224466234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6831","nonce":224466234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":224466234},{"type":"v1","addr":"172.21.15.157:6829","nonce":224466234}]},"public_addr":"172.21.15.157:6825/224466234","cluster_addr":"172.21.15.157:6827/224466234","heartbeat_back_addr":"172.21.15.157:6831/224466234","heartbeat_front_addr":"172.21.15.157:6829/224466234","state":["exists","up"]},{"osd":7,"uuid":"6736762c-62fd-4145-8a2d-91f485a23404","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6811","nonce":498117293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6815","nonce":498117293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6823","nonce":498117293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":498117293},{"type":"v1","addr":"172.21.15.157:6819","nonce":498117293}]},"public_addr":"172.21.15.157:6811/498117293","cluster_addr":"172.21.15.157:6815/498117293","heartbeat_back_addr":"172.21.15.157:6823/498117293","heartbeat_front_addr":"172.21.15.157:6819/498117293","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.086232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.334473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.323000+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.303314+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.249877+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.145681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.196906+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T11:20:02.272826+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6837/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/1501623433":"2024-04-26T11:41:59.046316+0000","172.21.15.157:6835/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3576995854":"2024-04-26T11:36:58.974013+0000","172.21.15.157:6837/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6836/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6834/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6838/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6835/430807169":"2024-04-26T11:23:23.787444+0000","172.21.15.157:6839/3020784477":"2024-04-26T11:42:54.061351+0000","172.21.15.136:6833/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.136:6836/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.136:6832/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6834/3463942074":"2024-04-26T11:38:18.992149+0000","172.21.15.136:6832/3353376405":"2024-04-26T11:25:58.820255+0000","172.21.15.157:6839/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6834/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.136:6836/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.136:6837/1740633057":"2024-04-26T11:44:19.077214+0000","172.21.15.157:6838/1671830331":"2024-04-26T11:32:38.912872+0000","172.21.15.136:6837/1368493675":"2024-04-26T11:28:33.853864+0000","172.21.15.157:6835/3688751078":"2024-04-26T11:33:28.923953+0000","172.21.15.157:6835/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6833/906354453":"2024-04-26T11:39:24.008268+0000","172.21.15.157:6837/281980057":"2024-04-26T11:30:08.877388+0000","172.21.15.136:6835/3406304193":"2024-04-26T11:35:58.955925+0000","172.21.15.157:6834/694883637":"2024-04-26T11:30:48.887575+0000","172.21.15.157:6836/694883637":"2024-04-26T11:30:48.887575+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 msgr2=0x7f8c6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 0x7f8c6c071910 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009a00 tx=0x7f8c84008040 comp rx=0 tx=0).stop 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 msgr2=0x7f8c880cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c000fe0 tx=0x7f8c7c00dc00 comp rx=0 tx=0).stop 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 shutdown_connections 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f8c6c06f400 0x7f8c6c071910 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8c881e5eb0 0x7f8c881e62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.743 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c8814b4d0 0x7f8c880cbb70 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 --2- 172.21.15.136:0/2386609506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c74004d50 0x7f8c881f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 >> 172.21.15.136:0/2386609506 conn(0x7f8c880ca5a0 msgr2=0x7f8c8814d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 shutdown_connections 2024-04-25T11:45:52.744 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.759+0000 7f8c8fb8f640 1 -- 172.21.15.136:0/2386609506 wait complete. 2024-04-25T11:45:52.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T11:45:52.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.927+0000 7f185d3dd640 1 Processor -- start 2024-04-25T11:45:52.914 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f185d3dd640 1 -- start start 2024-04-25T11:45:52.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f185d3dd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f185806cff0 0x7f1858163d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f185d3dd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1858152aa0 con 0x7f1858152690 2024-04-25T11:45:52.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f185d3dd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18580c3b10 con 0x7f18581583c0 2024-04-25T11:45:52.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f185d3dd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f18580c3df0 con 0x7f185806cff0 2024-04-25T11:45:52.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1858152690 0x7f185806c8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:42164/0 (socket says 172.21.15.136:42164) 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 -- 172.21.15.136:0/1101771293 learned_addr learned my addr 172.21.15.136:0/1101771293 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18577fe640 1 --2- 172.21.15.136:0/1101771293 >> v2:172.21.15.157:3300/0 conn(0x7f185806cff0 0x7f1858163d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090845571 0 0) 0x7f1858152aa0 con 0x7f1858152690 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1844003530 con 0x7f1858152690 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1825444789 0 0) 0x7f1844003530 con 0x7f1858152690 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 >> v2:172.21.15.157:3300/0 conn(0x7f185806cff0 msgr2=0x7f1858163d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 --2- 172.21.15.136:0/1101771293 >> v2:172.21.15.157:3300/0 conn(0x7f185806cff0 0x7f1858163d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 >> v1:172.21.15.157:6789/0 conn(0x7f18581583c0 legacy=0x7f18580c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18580c40d0 con 0x7f1858152690 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f1840001680 con 0x7f1858152690 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 >> v1:172.21.15.136:6789/0 conn(0x7f1858152690 legacy=0x7f185806c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440039c0 0x7f1844003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 0x7f1844004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18440051d0 0x7f184400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18580c3df0 con 0x7f1844004810 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f184400c160 con 0x7f18440051d0 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f184400c360 con 0x7f18440039c0 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 0x7f1844004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18577fe640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18440051d0 0x7f184400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f1856ffd640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440039c0 0x7f1844003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 -- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440039c0 msgr2=0x7f1844003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440039c0 0x7f1844003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 -- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18440051d0 msgr2=0x7f184400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18440051d0 0x7f184400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f18577fe640 1 -- 172.21.15.136:0/1101771293 reap_dead start 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f184e7fc640 1 -- 172.21.15.136:0/1101771293 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1844003530 con 0x7f1844004810 2024-04-25T11:45:52.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.931+0000 7f1856ffd640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f18440039c0 0x7f1844003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:52.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f184e7fc640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 0x7f1844004c20 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f184000c240 tx=0x7f184001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe8d8cec043bcf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840007d20 con 0x7f1844004810 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f18400a9070 con 0x7f1844004810 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f18567fc640 1 -- 172.21.15.136:0/1101771293 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18400a5e20 con 0x7f1844004810 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 -- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 msgr2=0x7f1844004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 0x7f1844004c20 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f184000c240 tx=0x7f184001cc40 comp rx=0 tx=0).stop 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 -- 172.21.15.136:0/1101771293 shutdown_connections 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 --2- 172.21.15.136:0/1101771293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1844004810 0x7f1844004c20 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 -- 172.21.15.136:0/1101771293 >> 172.21.15.136:0/1101771293 conn(0x7f185806f730 msgr2=0x7f185806fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 -- 172.21.15.136:0/1101771293 shutdown_connections 2024-04-25T11:45:52.922 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.935+0000 7f185d3dd640 1 -- 172.21.15.136:0/1101771293 wait complete. 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 Processor -- start 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 -- start start 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1844004810 0x7f18581f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18580687d0 0x7f18580cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18580c4970 con 0x7f18580cc3c0 2024-04-25T11:45:52.927 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f18580c4690 con 0x7f18580687d0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f185d3dd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f18580ca2e0 con 0x7f1844004810 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38574/0 (socket says 172.21.15.136:38574) 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 -- 172.21.15.136:0/593672159 learned_addr learned my addr 172.21.15.136:0/593672159 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f1856ffd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1844004810 0x7f18581f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f18577fe640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18580687d0 0x7f18580cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 -- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1844004810 msgr2=0x7f18581f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1844004810 0x7f18581f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 -- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18580687d0 msgr2=0x7f18580cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18580687d0 0x7f18580cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.939+0000 7f184e7fc640 1 -- 172.21.15.136:0/593672159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18581f1270 con 0x7f18580cc3c0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184e7fc640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f184001ca70 tx=0x7f1844005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18400bc070 con 0x7f18580cc3c0 2024-04-25T11:45:52.928 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1840000ee0 con 0x7f18580cc3c0 2024-04-25T11:45:52.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185813dc20 con 0x7f18580cc3c0 2024-04-25T11:45:52.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840007d20 con 0x7f18580cc3c0 2024-04-25T11:45:52.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581f0530 con 0x7f18580cc3c0 2024-04-25T11:45:52.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185813e140 con 0x7f18580cc3c0 2024-04-25T11:45:52.929 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f18400bf070 con 0x7f18580cc3c0 2024-04-25T11:45:52.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 0x7f1838071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:52.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7939+0+0 (secure 0 0 0) 0x7f1840129480 con 0x7f18580cc3c0 2024-04-25T11:45:52.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f1856ffd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 0x7f1838071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:52.930 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.943+0000 7f1856ffd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 0x7f1838071940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1858051a60 tx=0x7f1848009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:52.933 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:52.947+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f185813e140 con 0x7f18580cc3c0 2024-04-25T11:45:53.131 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.147+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f185813d9b0 con 0x7f18580cc3c0 2024-04-25T11:45:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.147+0000 7f184ffff640 1 -- 172.21.15.136:0/593672159 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) ==== 161+0+0 (secure 0 0 0) 0x7f1840016950 con 0x7f18580cc3c0 2024-04-25T11:45:53.134 INFO:teuthology.orchestra.run.smithi136.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T11:45:53.140 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 msgr2=0x7f1838071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 0x7f1838071940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1858051a60 tx=0x7f1848009040 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 msgr2=0x7f18581f0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f184001ca70 tx=0x7f1844005840 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 shutdown_connections 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18580687d0 0x7f18580cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18580cc3c0 0x7f18581f0d30 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f183806f430 0x7f1838071940 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 --2- 172.21.15.136:0/593672159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1844004810 0x7f18581f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 >> 172.21.15.136:0/593672159 conn(0x7f185806f730 msgr2=0x7f185806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 shutdown_connections 2024-04-25T11:45:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.155+0000 7f185d3dd640 1 -- 172.21.15.136:0/593672159 wait complete. 2024-04-25T11:45:53.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T11:45:53.213 INFO:tasks.ceph.mds.c.smithi136.stdout:starting mds.c at 2024-04-25T11:45:53.214 INFO:tasks.ceph.mds.a.smithi136.stdout:starting mds.a at 2024-04-25T11:45:53.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 Processor -- start 2024-04-25T11:45:53.308 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 -- start start 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efdb40b5060 0x7efdb40c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb415d750 con 0x7efdb415ab40 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb415da30 con 0x7efdb40ca290 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb94fb640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efdb415dd10 con 0x7efdb40b5060 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7efdb415ab40 0x7efdb4164780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:42192/0 (socket says 172.21.15.136:42192) 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb37fe640 1 -- 172.21.15.136:0/969555210 learned_addr learned my addr 172.21.15.136:0/969555210 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028939838 0 0) 0x7efdb415d750 con 0x7efdb415ab40 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd9c003530 con 0x7efdb415ab40 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb2ffd640 1 --2- 172.21.15.136:0/969555210 >> v2:172.21.15.157:3300/0 conn(0x7efdb40b5060 0x7efdb40c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.311 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1728624737 0 0) 0x7efd9c003530 con 0x7efdb415ab40 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 >> v2:172.21.15.157:3300/0 conn(0x7efdb40b5060 msgr2=0x7efdb40c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 --2- 172.21.15.136:0/969555210 >> v2:172.21.15.157:3300/0 conn(0x7efdb40b5060 0x7efdb40c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 >> v1:172.21.15.157:6789/0 conn(0x7efdb40ca290 legacy=0x7efdb415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb415dff0 con 0x7efdb415ab40 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7efda4001a00 con 0x7efdb415ab40 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 >> v1:172.21.15.136:6789/0 conn(0x7efdb415ab40 legacy=0x7efdb4164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd9c0039c0 0x7efd9c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.323+0000 7efdb1ffb640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd9c004810 0x7efd9c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efd9c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efdb415dd10 con 0x7efd9c005280 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c00c200 con 0x7efd9c004810 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd9c00c400 con 0x7efd9c0039c0 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efd9c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.312 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb27fc640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd9c004810 0x7efd9c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb2ffd640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd9c0039c0 0x7efd9c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 -- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd9c0039c0 msgr2=0x7efd9c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd9c0039c0 0x7efd9c003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 -- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd9c004810 msgr2=0x7efd9c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd9c004810 0x7efd9c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.313 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 -- 172.21.15.136:0/969555210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9c003530 con 0x7efd9c005280 2024-04-25T11:45:53.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 -- 172.21.15.136:0/969555210 reap_dead start 2024-04-25T11:45:53.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb37fe640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efd9c00b9d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7efdb4072500 tx=0x7efda401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a593c93e7e4088 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda400a4a0 con 0x7efd9c005280 2024-04-25T11:45:53.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efda40ad650 con 0x7efd9c005280 2024-04-25T11:45:53.314 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb1ffb640 1 -- 172.21.15.136:0/969555210 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda40ada60 con 0x7efd9c005280 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 -- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 msgr2=0x7efd9c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efd9c00b9d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7efdb4072500 tx=0x7efda401d340 comp rx=0 tx=0).stop 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 -- 172.21.15.136:0/969555210 shutdown_connections 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 --2- 172.21.15.136:0/969555210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efd9c00b9d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 -- 172.21.15.136:0/969555210 >> 172.21.15.136:0/969555210 conn(0x7efdb4070f70 msgr2=0x7efdb4068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 -- 172.21.15.136:0/969555210 shutdown_connections 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.327+0000 7efdb94fb640 1 -- 172.21.15.136:0/969555210 wait complete. 2024-04-25T11:45:53.315 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 Processor -- start 2024-04-25T11:45:53.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- start start 2024-04-25T11:45:53.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb40cd300 0x7efdb415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb415b6d0 0x7efdb41f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efdb415d3f0 con 0x7efd9c005280 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efdb4159f10 con 0x7efdb415b6d0 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efdb415d570 con 0x7efdb40cd300 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38622/0 (socket says 172.21.15.136:38622) 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 -- 172.21.15.136:0/2324259743 learned_addr learned my addr 172.21.15.136:0/2324259743 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 -- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb40cd300 msgr2=0x7efdb415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb40cd300 0x7efdb415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 -- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb415b6d0 msgr2=0x7efdb41f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb415b6d0 0x7efdb41f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:53.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 -- 172.21.15.136:0/2324259743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb40c5410 con 0x7efd9c005280 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb2ffd640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7efdac00e610 tx=0x7efdac00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac017070 con 0x7efd9c005280 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efdac013610 con 0x7efd9c005280 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac00f040 con 0x7efd9c005280 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb40cdf00 con 0x7efd9c005280 2024-04-25T11:45:53.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.331+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb40ce4c0 con 0x7efd9c005280 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb406a350 con 0x7efd9c005280 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efdac01a070 con 0x7efd9c005280 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efd9b7fe640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 0x7efda0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 8369+0+0 (secure 0 0 0) 0x7efdac08fc70 con 0x7efd9c005280 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efdb27fc640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 0x7efda0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:53.320 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.335+0000 7efdb27fc640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 0x7efda0071940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efda8001500 tx=0x7efda8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:53.323 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.339+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efdac05c120 con 0x7efd9c005280 2024-04-25T11:45:53.384 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-04-25T11:45:53.388 INFO:tasks.ceph.mds.f.smithi157.stdout:starting mds.f at 2024-04-25T11:45:53.394 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-04-25T11:45:53.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:53.535+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efdb40c5de0 con 0x7efd9c005280 2024-04-25T11:45:54.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.171+0000 7efd9b7fe640 1 -- 172.21.15.136:0/2324259743 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) ==== 104+0+0 (secure 0 0 0) 0x7efdac05ba20 con 0x7efd9c005280 2024-04-25T11:45:54.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.171+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 msgr2=0x7efda0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.171+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 0x7efda0071940 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efda8001500 tx=0x7efda8009040 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.171+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 msgr2=0x7efdb414ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.171+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7efdac00e610 tx=0x7efdac00ead0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 shutdown_connections 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdb415b6d0 0x7efdb41f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7efda006f430 0x7efda0071940 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdb40cd300 0x7efdb415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 --2- 172.21.15.136:0/2324259743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd9c005280 0x7efdb414ef50 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 >> 172.21.15.136:0/2324259743 conn(0x7efdb4070f70 msgr2=0x7efdb40cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 shutdown_connections 2024-04-25T11:45:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.175+0000 7efdb94fb640 1 -- 172.21.15.136:0/2324259743 wait complete. 2024-04-25T11:45:54.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T11:45:54.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 Processor -- start 2024-04-25T11:45:54.324 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 -- start start 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d600cb510 0x7f3d600c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3d600c7fe0 0x7f3d60163820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d60074ad0 con 0x7f3d600cb100 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f3d60074d90 con 0x7f3d600cb510 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d653d6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3d600c5890 con 0x7f3d600c7fe0 2024-04-25T11:45:54.326 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d600cb510 0x7f3d600c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d600cb510 0x7f3d600c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38632/0 (socket says 172.21.15.136:38632) 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 -- 172.21.15.136:0/4136789291 learned_addr learned my addr 172.21.15.136:0/4136789291 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5f7fe640 1 --2- 172.21.15.136:0/4136789291 >> v2:172.21.15.157:3300/0 conn(0x7f3d600c7fe0 0x7f3d60163820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 -- 172.21.15.136:0/4136789291 >> v2:172.21.15.157:3300/0 conn(0x7f3d600c7fe0 msgr2=0x7f3d60163820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 --2- 172.21.15.136:0/4136789291 >> v2:172.21.15.157:3300/0 conn(0x7f3d600c7fe0 0x7f3d60163820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 -- 172.21.15.136:0/4136789291 >> v1:172.21.15.157:6789/0 conn(0x7f3d600cb100 legacy=0x7f3d60152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:54.327 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.339+0000 7f3d5e7fc640 1 -- 172.21.15.136:0/4136789291 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d600c5b70 con 0x7f3d600cb510 2024-04-25T11:45:54.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d5e7fc640 1 --2- 172.21.15.136:0/4136789291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d600c7aa0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3d600c8900 tx=0x7f3d5401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7587ea9586561084 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.328 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d5dffb640 1 -- 172.21.15.136:0/4136789291 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540b0070 con 0x7f3d600cb510 2024-04-25T11:45:54.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d5dffb640 1 -- 172.21.15.136:0/4136789291 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3d540ab600 con 0x7f3d600cb510 2024-04-25T11:45:54.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d5dffb640 1 -- 172.21.15.136:0/4136789291 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540ab9f0 con 0x7f3d600cb510 2024-04-25T11:45:54.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 -- 172.21.15.136:0/4136789291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 msgr2=0x7f3d600c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 --2- 172.21.15.136:0/4136789291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d600c7aa0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3d600c8900 tx=0x7f3d5401c450 comp rx=0 tx=0).stop 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 -- 172.21.15.136:0/4136789291 shutdown_connections 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 --2- 172.21.15.136:0/4136789291 >> v2:172.21.15.157:3300/0 conn(0x7f3d600c7fe0 0x7f3d60163820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 --2- 172.21.15.136:0/4136789291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d600c7aa0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 -- 172.21.15.136:0/4136789291 >> 172.21.15.136:0/4136789291 conn(0x7f3d6006d950 msgr2=0x7f3d6006dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 -- 172.21.15.136:0/4136789291 shutdown_connections 2024-04-25T11:45:54.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.343+0000 7f3d653d6640 1 -- 172.21.15.136:0/4136789291 wait complete. 2024-04-25T11:45:54.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 Processor -- start 2024-04-25T11:45:54.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- start start 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 0x7f3d60153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d600cb100 0x7f3d60155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600caa60 con 0x7f3d600cb510 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600cb920 con 0x7f3d600c7fe0 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3d600cabe0 con 0x7f3d600cb100 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:38648/0 (socket says 172.21.15.136:38648) 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 -- 172.21.15.136:0/889396066 learned_addr learned my addr 172.21.15.136:0/889396066 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5effd640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 0x7f3d60153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 -- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d600cb100 msgr2=0x7f3d60155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d600cb100 0x7f3d60155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 -- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 msgr2=0x7f3d60153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 0x7f3d60153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 -- 172.21.15.136:0/889396066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d60152d30 con 0x7f3d600cb510 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5effd640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 0x7f3d60153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d5f7fe640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3d50007e90 tx=0x7f3d5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50014420 con 0x7f3d600cb510 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3d50019070 con 0x7f3d600cb510 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d60153fc0 con 0x7f3d600cb510 2024-04-25T11:45:54.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50014820 con 0x7f3d600cb510 2024-04-25T11:45:54.336 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.347+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d60153280 con 0x7f3d600cb510 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d60159da0 con 0x7f3d600cb510 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d500149c0 con 0x7f3d600cb510 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d477fe640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 0x7f3d4c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 8369+0+0 (secure 0 0 0) 0x7f3d500906b0 con 0x7f3d600cb510 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d5effd640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 0x7f3d4c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:54.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.351+0000 7f3d5effd640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 0x7f3d4c071940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800a850 tx=0x7f3d58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:54.341 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.355+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d5005cae0 con 0x7f3d600cb510 2024-04-25T11:45:54.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:54.507+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3d60153770 con 0x7f3d600cb510 2024-04-25T11:45:55.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.183+0000 7f3d477fe640 1 -- 172.21.15.136:0/889396066 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v42) ==== 181+0+0 (secure 0 0 0) 0x7f3d5005c3e0 con 0x7f3d600cb510 2024-04-25T11:45:55.167 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_metadata' removed 2024-04-25T11:45:55.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.183+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 msgr2=0x7f3d4c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.183+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 0x7f3d4c071940 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800a850 tx=0x7f3d58008040 comp rx=0 tx=0).stop 2024-04-25T11:45:55.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.183+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 msgr2=0x7f3d601f8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.183+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3d50007e90 tx=0x7f3d5000e4b0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 shutdown_connections 2024-04-25T11:45:55.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d600cb510 0x7f3d601f8440 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f3d4c06f430 0x7f3d4c071940 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3d600cb100 0x7f3d60155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 --2- 172.21.15.136:0/889396066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d600c7fe0 0x7f3d60153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.186+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 >> 172.21.15.136:0/889396066 conn(0x7f3d6006d950 msgr2=0x7f3d6006c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.190+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 shutdown_connections 2024-04-25T11:45:55.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.190+0000 7f3d653d6640 1 -- 172.21.15.136:0/889396066 wait complete. 2024-04-25T11:45:55.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T11:45:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 Processor -- start 2024-04-25T11:45:55.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 -- start start 2024-04-25T11:45:55.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56680cb920 0x7f56680cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f56680cc270 0x7f5668166aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5668156960 con 0x7f5668156550 2024-04-25T11:45:55.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 -- 0x7f566815cc80 con 0x7f56680cb920 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f566d734640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f566815d1b0 con 0x7f56680cc270 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56680cb920 0x7f56680cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f56680cb920 0x7f56680cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44672/0 (socket says 172.21.15.136:44672) 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 -- 172.21.15.136:0/1133538112 learned_addr learned my addr 172.21.15.136:0/1133538112 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56677fe640 1 --2- 172.21.15.136:0/1133538112 >> v2:172.21.15.157:3300/0 conn(0x7f56680cc270 0x7f5668166aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 -- 172.21.15.136:0/1133538112 >> v2:172.21.15.157:3300/0 conn(0x7f56680cc270 msgr2=0x7f5668166aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 --2- 172.21.15.136:0/1133538112 >> v2:172.21.15.157:3300/0 conn(0x7f56680cc270 0x7f5668166aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 -- 172.21.15.136:0/1133538112 >> v1:172.21.15.157:6789/0 conn(0x7f5668156550 legacy=0x7f56680703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:55.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.362+0000 7f56667fc640 1 -- 172.21.15.136:0/1133538112 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566815ced0 con 0x7f56680cb920 2024-04-25T11:45:55.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f56667fc640 1 --2- 172.21.15.136:0/1133538112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56680cb920 0x7f56680cbd30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f56680cd050 tx=0x7f565001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec64c41be5af405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f5665ffb640 1 -- 172.21.15.136:0/1133538112 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500b0070 con 0x7f56680cb920 2024-04-25T11:45:55.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f5665ffb640 1 -- 172.21.15.136:0/1133538112 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f56500ab600 con 0x7f56680cb920 2024-04-25T11:45:55.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f5665ffb640 1 -- 172.21.15.136:0/1133538112 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500ab9f0 con 0x7f56680cb920 2024-04-25T11:45:55.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 -- 172.21.15.136:0/1133538112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56680cb920 msgr2=0x7f56680cbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 --2- 172.21.15.136:0/1133538112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56680cb920 0x7f56680cbd30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f56680cd050 tx=0x7f565001c450 comp rx=0 tx=0).stop 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 -- 172.21.15.136:0/1133538112 shutdown_connections 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 --2- 172.21.15.136:0/1133538112 >> v2:172.21.15.157:3300/0 conn(0x7f56680cc270 0x7f5668166aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 --2- 172.21.15.136:0/1133538112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56680cb920 0x7f56680cbd30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 -- 172.21.15.136:0/1133538112 >> 172.21.15.136:0/1133538112 conn(0x7f56680726d0 msgr2=0x7f5668072ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 -- 172.21.15.136:0/1133538112 shutdown_connections 2024-04-25T11:45:55.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.366+0000 7f566d734640 1 -- 172.21.15.136:0/1133538112 wait complete. 2024-04-25T11:45:55.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.370+0000 7f566d734640 1 Processor -- start 2024-04-25T11:45:55.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.370+0000 7f566d734640 1 -- start start 2024-04-25T11:45:55.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.370+0000 7f566d734640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56680cb920 0x7f566815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 0x7f56680cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f56680cc800 con 0x7f5668156550 2024-04-25T11:45:55.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f56680cc680 con 0x7f56680cc270 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f56680b20e0 con 0x7f56680cb920 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44678/0 (socket says 172.21.15.136:44678) 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 -- 172.21.15.136:0/2204286992 learned_addr learned my addr 172.21.15.136:0/2204286992 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 -- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56680cb920 msgr2=0x7f566815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56667fc640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 0x7f56680cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f5666ffd640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56680cb920 0x7f566815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56680cb920 0x7f566815e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 -- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 msgr2=0x7f56680cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 0x7f56680cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:55.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 -- 172.21.15.136:0/2204286992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56680cfe30 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56667fc640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 0x7f56680cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f56677fe640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f566000cc00 tx=0x7f566000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566000ce10 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f56600039a0 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56680b1300 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660003d90 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5668153960 con 0x7f5668156550 2024-04-25T11:45:55.362 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566806a3c0 con 0x7f5668156550 2024-04-25T11:45:55.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5660003f30 con 0x7f5668156550 2024-04-25T11:45:55.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 0x7f5648071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:55.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7931+0+0 (secure 0 0 0) 0x7f5660090030 con 0x7f5668156550 2024-04-25T11:45:55.363 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.374+0000 7f5666ffd640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 0x7f5648071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:55.364 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.378+0000 7f5666ffd640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 0x7f5648071910 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f565800a750 tx=0x7f5658008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:55.366 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.378+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5660014030 con 0x7f5668156550 2024-04-25T11:45:55.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:55.558+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f566806dab0 con 0x7f5668156550 2024-04-25T11:45:56.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.190+0000 7f565f7fe640 1 -- 172.21.15.136:0/2204286992 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v43) ==== 169+0+0 (secure 0 0 0) 0x7f566005c670 con 0x7f5668156550 2024-04-25T11:45:56.176 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_data' removed 2024-04-25T11:45:56.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.202+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 msgr2=0x7f5648071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.202+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 0x7f5648071910 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f565800a750 tx=0x7f5658008040 comp rx=0 tx=0).stop 2024-04-25T11:45:56.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.202+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 msgr2=0x7f56680c52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.202+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f566000cc00 tx=0x7f566000a5c0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 shutdown_connections 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5668156550 0x7f56680c52e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f56680cc270 0x7f56680cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f564806f400 0x7f5648071910 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 --2- 172.21.15.136:0/2204286992 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f56680cb920 0x7f566815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 >> 172.21.15.136:0/2204286992 conn(0x7f56680726d0 msgr2=0x7f566815b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 shutdown_connections 2024-04-25T11:45:56.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.206+0000 7f566d734640 1 -- 172.21.15.136:0/2204286992 wait complete. 2024-04-25T11:45:56.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T11:45:56.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 Processor -- start 2024-04-25T11:45:56.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 -- start start 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f27e00c4410 0x7f27e00c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27e015df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e0071cb0 con 0x7f27e00cb2c0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f27e0072230 con 0x7f27e00c4410 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e7d4e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27e00cb6d0 con 0x7f27e015daf0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e5aea640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f27e00cb2c0 0x7f27e01540e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:44458/0 (socket says 172.21.15.136:44458) 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e5aea640 1 -- 172.21.15.136:0/1765163176 learned_addr learned my addr 172.21.15.136:0/1765163176 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e62eb640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27e015df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 --2- 172.21.15.136:0/1765163176 >> v2:172.21.15.157:3300/0 conn(0x7f27e00c4410 0x7f27e00c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537839966 0 0) 0x7f27e0071cb0 con 0x7f27e00cb2c0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27cc003530 con 0x7f27e00cb2c0 2024-04-25T11:45:56.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2266223990 0 0) 0x7f27cc003530 con 0x7f27e00cb2c0 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 msgr2=0x7f27e015df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27e015df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 >> v2:172.21.15.157:3300/0 conn(0x7f27e00c4410 msgr2=0x7f27e00c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 --2- 172.21.15.136:0/1765163176 >> v2:172.21.15.157:3300/0 conn(0x7f27e00c4410 0x7f27e00c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e015ab30 con 0x7f27e00cb2c0 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2601850825 0 0) 0x7f27dc001680 con 0x7f27e00cb2c0 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 >> v1:172.21.15.136:6789/0 conn(0x7f27e00cb2c0 legacy=0x7f27e01540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27cc0039c0 0x7f27cc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27cc004810 0x7f27cc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27cc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0072230 con 0x7f27cc004810 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27e00cb6d0 con 0x7f27cc0039c0 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc00ba80 con 0x7f27e015daf0 2024-04-25T11:45:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27cc004810 0x7f27cc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e5aea640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27cc0039c0 0x7f27cc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e62eb640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27cc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 -- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 msgr2=0x7f27cc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e015daf0 0x7f27cc005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 -- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27cc0039c0 msgr2=0x7f27cc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27cc0039c0 0x7f27cc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.386+0000 7f27e52e9640 1 -- 172.21.15.136:0/1765163176 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc003530 con 0x7f27cc004810 2024-04-25T11:45:56.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e52e9640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27cc004810 0x7f27cc004c20 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f27e0074a40 tx=0x7f27d001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efe833bf2bb6dda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:56.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00096b0 con 0x7f27cc004810 2024-04-25T11:45:56.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f27d00a6430 con 0x7f27cc004810 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e4ae8640 1 -- 172.21.15.136:0/1765163176 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00a6840 con 0x7f27cc004810 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 -- 172.21.15.136:0/1765163176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27cc004810 msgr2=0x7f27cc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/1765163176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27cc004810 0x7f27cc004c20 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f27e0074a40 tx=0x7f27d001ca50 comp rx=0 tx=0).stop 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e62eb640 1 -- 172.21.15.136:0/1765163176 reap_dead start 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 -- 172.21.15.136:0/1765163176 shutdown_connections 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 -- 172.21.15.136:0/1765163176 >> 172.21.15.136:0/1765163176 conn(0x7f27e0068cd0 msgr2=0x7f27e0072770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 -- 172.21.15.136:0/1765163176 shutdown_connections 2024-04-25T11:45:56.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.390+0000 7f27e7d4e640 1 -- 172.21.15.136:0/1765163176 wait complete. 2024-04-25T11:45:56.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 Processor -- start 2024-04-25T11:45:56.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- start start 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27e0147100 0x7f27e015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e0145ae0 0x7f27e0145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f27e00c4ff0 con 0x7f27e00c67f0 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27e015e310 con 0x7f27e0147100 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27e00c5170 con 0x7f27e0145ae0 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:44700/0 (socket says 172.21.15.136:44700) 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 -- 172.21.15.136:0/3805845515 learned_addr learned my addr 172.21.15.136:0/3805845515 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e5aea640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27e0147100 0x7f27e015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.382 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e52e9640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e0145ae0 0x7f27e0145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 -- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e0145ae0 msgr2=0x7f27e0145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e0145ae0 0x7f27e0145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 -- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27e0147100 msgr2=0x7f27e015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27e0147100 0x7f27e015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 -- 172.21.15.136:0/3805845515 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e0147e10 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e62eb640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f27e00746f0 tx=0x7f27d800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d801c070 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f27d8017400 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e0147a50 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8017810 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e00c8460 con 0x7f27e00c67f0 2024-04-25T11:45:56.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.394+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e006a3c0 con 0x7f27e00c67f0 2024-04-25T11:45:56.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.398+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f27d801f070 con 0x7f27e00c67f0 2024-04-25T11:45:56.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.398+0000 7f27d67fc640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 0x7f27c4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T11:45:56.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.398+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7401+0+0 (secure 0 0 0) 0x7f27d80904a0 con 0x7f27e00c67f0 2024-04-25T11:45:56.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.398+0000 7f27e5aea640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 0x7f27c4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T11:45:56.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.398+0000 7f27e5aea640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 0x7f27c4071940 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000fc0 tx=0x7f27dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T11:45:56.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.402+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f27d8026020 con 0x7f27e00c67f0 2024-04-25T11:45:56.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:56.562+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f27e0151a30 con 0x7f27e00c67f0 2024-04-25T11:45:57.185 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.198+0000 7f27d67fc640 1 -- 172.21.15.136:0/3805845515 <== mon.0 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v44) ==== 178+0+0 (secure 0 0 0) 0x7f27e0147a50 con 0x7f27e00c67f0 2024-04-25T11:45:57.185 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'cephfs_data_ec' removed 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 msgr2=0x7f27c4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 0x7f27c4071940 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000fc0 tx=0x7f27dc009040 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 msgr2=0x7f27e00c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f27e00746f0 tx=0x7f27d800f4a0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 shutdown_connections 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27e00c67f0 0x7f27e00c6c00 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:6832/602770275,v1:172.21.15.157:6833/602770275] conn(0x7f27c406f430 0x7f27c4071940 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27e0145ae0 0x7f27e0145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 --2- 172.21.15.136:0/3805845515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27e0147100 0x7f27e015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 >> 172.21.15.136:0/3805845515 conn(0x7f27e0068cd0 msgr2=0x7f27e0159470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 shutdown_connections 2024-04-25T11:45:57.189 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-25T11:45:57.202+0000 7f27e7d4e640 1 -- 172.21.15.136:0/3805845515 wait complete. 2024-04-25T11:45:57.215 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T11:45:57.216 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T11:45:57.216 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.233 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T11:45:57.233 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T11:45:57.233 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.324 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T11:45:57.324 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-25T11:45:57.324 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.343 INFO:tasks.ceph.mds.e:Stopped 2024-04-25T11:45:57.343 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T11:45:57.344 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.367 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T11:45:57.367 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T11:45:57.367 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.424 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T11:45:57.424 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-25T11:45:57.424 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:45:57.480 INFO:tasks.ceph.mds.f:Stopped 2024-04-25T11:45:57.480 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T11:45:57.480 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T11:45:57.480 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:01.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-25T11:46:03.581 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T11:46:03.582 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T11:46:03.582 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:07.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-25T11:46:07.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-25T11:46:09.684 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T11:46:09.685 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T11:46:09.685 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:09.776 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T11:46:09.777 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T11:46:09.777 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:13.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-25T11:46:13.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-25T11:46:15.878 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T11:46:15.879 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T11:46:15.879 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:19.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-25T11:46:19.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-25T11:46:21.981 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T11:46:21.981 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T11:46:21.981 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:22.080 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T11:46:22.080 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T11:46:22.080 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:24.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-25T11:46:24.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-25T11:46:28.182 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T11:46:28.182 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T11:46:28.182 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:30.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-25T11:46:30.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-25T11:46:34.284 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T11:46:34.285 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T11:46:34.285 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T11:46:34.285 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:34.313 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T11:46:34.313 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T11:46:34.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:34.348 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T11:46:34.348 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T11:46:34.349 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T11:46:34.349 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:34.377 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T11:46:34.377 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T11:46:34.378 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:34.415 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T11:46:34.415 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T11:46:34.415 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T11:46:34.490 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T11:46:34.491 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T11:46:34.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T11:46:34.661 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:46:34.661 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T11:46:35.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-25T11:46:36.247 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:46:36.249 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T11:46:36.542 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:46:36.543 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T11:46:36.811 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T11:46:36.811 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T11:46:37.059 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:46:37.059 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T11:46:37.365 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:46:37.365 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T11:46:37.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:46:37.627 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T11:46:37.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T11:46:37.813 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T11:46:38.015 INFO:tasks.ceph:Archiving mon data... 2024-04-25T11:46:38.016 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/data/mon.a.tgz 2024-04-25T11:46:38.016 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-04-25T11:46:38.047 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.oJLkjZ0eMi 2024-04-25T11:46:38.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oJLkjZ0eMi 2024-04-25T11:46:38.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.oJLkjZ0eMi 2024-04-25T11:46:38.309 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.oJLkjZ0eMi is 383KB 2024-04-25T11:46:38.367 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.oJLkjZ0eMi 2024-04-25T11:46:38.372 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/data/mon.b.tgz 2024-04-25T11:46:38.373 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-04-25T11:46:38.389 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.6StK9hREPI 2024-04-25T11:46:38.389 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6StK9hREPI 2024-04-25T11:46:38.535 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.6StK9hREPI 2024-04-25T11:46:38.634 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.6StK9hREPI is 390KB 2024-04-25T11:46:38.694 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.6StK9hREPI 2024-04-25T11:46:38.701 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/data/mon.c.tgz 2024-04-25T11:46:38.701 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-04-25T11:46:38.749 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.HcpQMrrRg0 2024-04-25T11:46:38.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.HcpQMrrRg0 2024-04-25T11:46:38.886 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.HcpQMrrRg0 2024-04-25T11:46:38.953 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.HcpQMrrRg0 is 390KB 2024-04-25T11:46:39.013 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.HcpQMrrRg0 2024-04-25T11:46:39.018 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T11:46:39.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T11:46:39.021 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T11:46:39.112 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T11:46:39.141 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T11:46:39.141 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T11:46:39.143 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi136/crash 2024-04-25T11:46:39.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T11:46:39.166 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi157/crash 2024-04-25T11:46:39.167 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T11:46:39.196 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T11:47:21.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T11:47:21.712 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T11:47:21.727 INFO:tasks.ceph:Compressing logs... 2024-04-25T11:47:21.728 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T11:47:21.759 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T11:47:21.779 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39489.log 2024-04-25T11:47:21.780 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60247.log 2024-04-25T11:47:21.780 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34864.log 2024-04-25T11:47:21.780 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43502.log 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54376.log 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60247.log.gz 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43817.log 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43502.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54376.log: /var/log/ceph/ceph-client.admin.35134.log 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43817.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54376.log.gz 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-04-25T11:47:21.782 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35134.log: -- /var/log/ceph/ceph-client.admin.60526.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61216.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60526.log.gz 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33374.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41737.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43954.log 2024-04-25T11:47:21.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54100.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35100.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44856.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35100.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54100.log.gz 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35643.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44040.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44882.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35643.log: /var/log/ceph/ceph-client.admin.44040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43344.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43502.log.gz 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41419.log: /var/log/ceph/ceph-client.admin.35743.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42408.log 2024-04-25T11:47:21.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27684.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27780.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27828.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27684.log: /var/log/ceph/ceph-client.admin.27780.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27852.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T11:47:21.805 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27852.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27804.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27708.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27756.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27708.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27756.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27780.log.gz 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-mds.f.log: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27828.log.gz 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T11:47:21.806 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27732.log 2024-04-25T11:47:21.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.4.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27804.log.gz 2024-04-25T11:47:21.807 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.27732.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27852.log.gz 2024-04-25T11:47:21.809 INFO:teuthology.orchestra.run.smithi157.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27684.log.gz 2024-04-25T11:47:21.809 INFO:teuthology.orchestra.run.smithi157.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27708.log.gz 2024-04-25T11:47:21.809 INFO:teuthology.orchestra.run.smithi157.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27732.log.gz 2024-04-25T11:47:21.843 INFO:teuthology.orchestra.run.smithi157.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T11:47:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42382.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40901.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42104.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33504.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36294.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60943.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40310.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60943.log.gz 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54350.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41471.log: /var/log/ceph/ceph-osd.0.log 2024-04-25T11:47:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60090.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54350.log.gz 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34027.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33923.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33158.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60090.log.gz 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61022.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35196.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33268.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35196.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61022.log.gz 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35538.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35538.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22755.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.tmp-client.admin.22755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22755.log.gz 2024-04-25T11:47:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34372.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47632.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43093.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34450.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.47632.log: /var/log/ceph/ceph-client.admin.43093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60351.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33792.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40360.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33766.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60351.log.gz 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36450.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27781.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60142.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.27781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27679.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60142.log.gz 2024-04-25T11:47:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.27679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43318.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43954.log.gz 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43900.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42700.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60998.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36060.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41867.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34864.log.gz 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36060.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60998.log.gz 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60116.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35100.log.gz 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41867.log: /var/log/ceph/ceph-client.admin.60116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43015.log 2024-04-25T11:47:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43844.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60116.log.gz 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35196.log.gz 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36554.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35643.log.gz 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41338.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36034.log: /var/log/ceph/ceph-client.admin.36554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44644.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35743.log.gz 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34868.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43874.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46753.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36294.log.gz 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.46753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61152.log 2024-04-25T11:47:21.875 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36450.log.gz 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44270.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35434.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33214.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43318.log.gz 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61152.log.gz 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33054.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41947.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33054.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36398.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59954.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34658.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43067.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59954.log.gz 2024-04-25T11:47:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39374.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55183.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34441.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33949.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55183.log.gz 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47362.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33348.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.47362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55878.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40563.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43292.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33844.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55878.log.gz 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44804.log 2024-04-25T11:47:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46997.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34448.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.46997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59847.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34448.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59379.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34136.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59847.log.gz 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33026.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39920.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59379.log.gz 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44460.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56200.log 2024-04-25T11:47:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34001.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43817.log.gz 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.56200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40794.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56200.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43844.log.gz 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43171.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34001.log: /var/log/ceph/ceph-client.admin.40794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33708.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43738.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/ceph-client.admin.33708.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34436.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36086.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41578.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34445.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41364.log 2024-04-25T11:47:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41364.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42540.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41656.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33132.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60474.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54247.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60474.log: /var/log/ceph/ceph-client.admin.41225.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60474.log.gz 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44538.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54247.log: /var/log/ceph/ceph-client.admin.44538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42278.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54247.log.gz 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33426.log 2024-04-25T11:47:21.880 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43712.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44672.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36320.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34640.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42883.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44698.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59981.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33080.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60604.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41763.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59981.log.gz 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60060.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60604.log.gz 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42025.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60299.log 2024-04-25T11:47:21.881 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42025.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.60060.log.gz -- 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr: /var/log/ceph/ceph-client.admin.59315.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57481.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60299.log.gz 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59315.log.gz 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.57481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40026.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40537.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57481.log.gz 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56676.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35793.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.56676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43660.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34451.log 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56676.log.gz 2024-04-25T11:47:21.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42805.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43344.log.gz 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27561.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43476.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.27561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35981.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45469.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.0.35981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35274.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.45469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39824.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43422.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46166.log 2024-04-25T11:47:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43422.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35134.log.gz 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.46166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33186.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35538.log.gz 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61100.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44301.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60552.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33000.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61100.log.gz 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55688.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60552.log: /var/log/ceph/ceph-client.admin.33000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42156.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60552.log.gz 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42224.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42156.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55688.log.gz 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59901.log 2024-04-25T11:47:21.884 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59794.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42224.log: gzip/var/log/ceph/ceph-client.admin.59901.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59794.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59901.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44192.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59794.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44069.log: /var/log/ceph/ceph-client.admin.44192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59768.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz/var/log/ceph/ceph-client.admin.41526.log: 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42596.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40794.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59768.log: /var/log/ceph/ceph-client.admin.42596.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59768.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46646.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36476.log 2024-04-25T11:47:21.885 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.46646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41973.log 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60194.log 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43067.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43093.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59728.log 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44040.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60194.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.33322.log 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44856.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59728.log: 88.8% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44882.log.gz -- replaced with /var/log/ceph/ceph-client.admin.60194.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44460.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59728.log.gz 2024-04-25T11:47:21.886 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36268.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32880.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43225.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.32880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44512.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42488.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44140.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56081.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36190.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.56081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60220.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36190.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56081.log.gz 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41789.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60220.log: /var/log/ceph/ceph-osd.1.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60220.log.gz 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2024-04-25T11:47:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60839.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41789.log: /var/log/ceph/ceph-client.admin.44908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60273.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34840.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60839.log.gz 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34847.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42436.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34840.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60273.log.gz 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42436.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40310.log.gz 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44095.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57620.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54428.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.57620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57620.log.gz 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39298.log 2024-04-25T11:47:21.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54428.log: /var/log/ceph/ceph-client.admin.39894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44830.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54428.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34084.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34268.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54454.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34053.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54508.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34001.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54454.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log/var/log/ceph/ceph-client.admin.40955.log: 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34372.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54321.log 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34448.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34436.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34450.log.gz 2024-04-25T11:47:21.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44592.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54508.log.gz 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34445.log.gz -- replaced with /var/log/ceph/ceph-client.admin.34441.log.gz 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54321.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54321.log.gz 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35512.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33296.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34110.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54480.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35460.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39515.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54480.log.gz 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43686.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35590.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39515.log: /var/log/ceph/ceph-client.admin.43686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34190.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44752.log 2024-04-25T11:47:21.890 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60422.log 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59586.log 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34644.log 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59586.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60422.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59586.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57070.log 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41815.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42382.log.gz 2024-04-25T11:47:21.891 INFO:teuthology.orchestra.run.smithi136.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43251.log 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.57070.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42408.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57070.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32916.log 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43900.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43660.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43874.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44192.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: -- /var/log/ceph/ceph-client.admin.34294.log/var/log/ceph/ceph-client.admin.32916.log: 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40360.log.gz 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43396.log 2024-04-25T11:47:21.892 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41630.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41630.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32880.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43015.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36112.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/ceph-client.admin.41173.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33530.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60578.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33530.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34868.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54402.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34659.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60578.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54402.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-04-25T11:47:21.893 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54402.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34659.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36086.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36060.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44564.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35274.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33478.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42330.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35326.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36112.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40731.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36268.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36398.log.gz 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43528.log 2024-04-25T11:47:21.894 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42462.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36476.log.gz 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43528.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36554.log.gz 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44407.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60734.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44355.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35074.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60734.log.gz 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gzgzip 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35074.log: /var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61126.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42831.log 2024-04-25T11:47:21.895 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44644.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61126.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44724.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61126.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45469.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34439.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47174.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41009.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.47174.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48206.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36164.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41009.log: /var/log/ceph/ceph-client.admin.48206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56442.log 2024-04-25T11:47:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36164.log: /var/log/ceph/ceph-client.admin.56442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42356.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56442.log.gz 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59875.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42937.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42356.log: /var/log/ceph/ceph-client.admin.59875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0%/var/log/ceph/ceph-client.admin.42937.log: -- replaced with /var/log/ceph/ceph-client.admin.59875.log.gz 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60500.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40703.log 2024-04-25T11:47:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.60500.log: /var/log/ceph/ceph-client.admin.60656.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40703.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60500.log.gz 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33870.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60656.log: /var/log/ceph/ceph-client.admin.33870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53845.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60656.log.gz 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60325.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.53845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35564.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33452.log 2024-04-25T11:47:21.972 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60325.log.gz 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/ceph-client.admin.35564.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35048.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43554.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.35460.log.gz 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34320.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35434.log.gz 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60682.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55074.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39868.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47362.log.gz 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55074.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60682.log.gz 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr: -- /var/log/ceph/ceph-client.admin.60865.log 2024-04-25T11:47:21.973 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55074.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39798.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60917.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60865.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35160.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39570.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60917.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60448.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41999.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36242.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60448.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41999.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35512.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36242.log: gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2024-04-25T11:47:21.974 INFO:teuthology.orchestra.run.smithi136.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39946.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39946.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36346.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.c.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57226.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47991.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.57226.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.47991.log: 87.1%gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.43767.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34136.log.gz 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57226.log.gz 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.43767.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42540.log.gz 89.3% 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61074.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41921.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43988.log 2024-04-25T11:47:21.975 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61074.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60812.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43988.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42622.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60812.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60708.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40026.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36528.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60708.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33975.log 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60708.log.gz/var/log/ceph/ceph-client.admin.36528.log: 2024-04-25T11:47:21.976 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40901.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: -- /var/log/ceph/ceph-client.admin.41303.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4%/var/log/ceph/ceph-client.admin.33975.log: -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41364.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35404.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41303.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44270.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42198.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33106.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42198.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43396.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43422.log.gz 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60760.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39701.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34164.log 2024-04-25T11:47:21.977 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60760.log.gz 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43634.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60168.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34867.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34867.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60168.log.gz 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61242.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60786.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59928.log 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60786.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61242.log.gz 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32823.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60786.log.gz 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43041.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.32823.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59928.log.gz 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54218.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59667.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54218.log.gz 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43928.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59667.log.gz 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61268.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54274.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43197.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54274.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61268.log.gz 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35693.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54274.log.gz 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-04-25T11:47:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42857.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42753.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61048.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44381.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.61048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33400.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61048.log.gz 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56900.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54150.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.56900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34398.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56900.log.gz 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42674.log 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54150.log.gz 2024-04-25T11:47:21.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34647.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60972.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60972.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60972.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35222.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42104.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55369.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43225.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43738.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33580.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55369.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45629.log 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2024-04-25T11:47:21.981 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43528.log.gz 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34858.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.45629.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44095.log.gz 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36424.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40981.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54036.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44327.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.54036.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 88.4% 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54036.log.gz 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44327.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34451.log.gz 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44014.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-04-25T11:47:21.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60630.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53558.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60630.log.gz 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.53558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41251.log: /var/log/ceph/ceph-client.admin.42077.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39868.log.gz 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35854.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39570.log.gz 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42648.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55531.log 2024-04-25T11:47:21.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59821.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42356.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42330.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44218.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42622.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.59821.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42831.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.55531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33242.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55531.log.gz 87.3% 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59821.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42857.log.gz 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41708.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.33242.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.43251.log.gz --verbose 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr: -- /var/log/ceph/ceph-client.admin.53656.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.41708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60891.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.53656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46441.log 2024-04-25T11:47:21.984 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/ceph-client.admin.60891.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44486.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.46441.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60891.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39544.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44486.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33949.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33923.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33975.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.39544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35486.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33504.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42726.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34848.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34084.log.gz 2024-04-25T11:47:21.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34848.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43145.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43607.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.42568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36502.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.43607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60377.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57371.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.45921.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33766.log.gz 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43370.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60377.log.gz 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.57371.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60010.log 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 87.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.57371.log.gz -- 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr: /var/log/ceph/ceph-client.admin.34216.log/var/log/ceph/ceph-client.admin.43370.log: 2024-04-25T11:47:21.986 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33792.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.60010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36372.log 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60010.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.34216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35300.log 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.36372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40178.log 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35300.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27679.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44244.log 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.40178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59519.log 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.44244.log: /var/log/ceph/ceph-client.admin.59519.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27781.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59519.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35300.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35326.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33026.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34840.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34858.log.gz 2024-04-25T11:47:21.987 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35048.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35074.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35854.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36242.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36424.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36502.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36528.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44752.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45629.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35160.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35404.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40981.log.gz 2024-04-25T11:47:21.988 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42156.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42198.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44672.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42596.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44014.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43988.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44486.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39701.log.gz 2024-04-25T11:47:21.989 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41630.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41867.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42488.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43292.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42436.log.gz 2024-04-25T11:47:21.990 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42648.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34027.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34053.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41947.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41973.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41999.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42278.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42700.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42726.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47174.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34190.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34110.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33580.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.33530.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-04-25T11:47:21.991 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33054.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39920.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33080.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43686.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43712.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34848.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34867.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.35793.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36190.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36346.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36164.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42753.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44804.log.gz 2024-04-25T11:47:21.992 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32823.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35564.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.35590.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42025.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44140.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39298.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40703.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41009.log.gz 2024-04-25T11:47:21.993 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44698.log.gz 2024-04-25T11:47:21.998 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39798.log.gz 2024-04-25T11:47:21.998 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39824.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42805.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42937.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44407.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44830.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41763.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44355.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44538.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-04-25T11:47:21.999 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40178.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33870.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34320.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43041.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34164.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39946.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43171.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43197.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43767.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33000.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34847.log.gz 2024-04-25T11:47:22.000 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36320.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.35693.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53845.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43928.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41921.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39374.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42883.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44301.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41578.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41708.log.gz 2024-04-25T11:47:22.001 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43607.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43634.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44564.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44381.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34644.log.gz 2024-04-25T11:47:22.002 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44327.log.gz 2024-04-25T11:47:22.007 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32916.log.gz 2024-04-25T11:47:22.008 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36372.log.gz 2024-04-25T11:47:22.010 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43476.log.gz 2024-04-25T11:47:22.010 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44724.log.gz 2024-04-25T11:47:22.012 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46441.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46166.log.gz 2024-04-25T11:47:22.012 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47632.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46997.log.gz 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46753.log.gz 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46646.log.gz 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47991.log.gz 2024-04-25T11:47:22.013 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48206.log.gz 2024-04-25T11:47:22.014 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35486.log.gz 2024-04-25T11:47:22.016 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-04-25T11:47:22.017 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42674.log.gz 2024-04-25T11:47:22.019 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53558.log.gz 2024-04-25T11:47:22.020 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53656.log.gz 2024-04-25T11:47:22.020 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-04-25T11:47:22.020 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40563.log.gz 2024-04-25T11:47:22.020 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43370.log.gz 2024-04-25T11:47:22.021 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42224.log.gz 2024-04-25T11:47:22.021 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44244.log.gz 2024-04-25T11:47:22.022 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-04-25T11:47:22.022 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-04-25T11:47:22.024 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43145.log.gz 2024-04-25T11:47:22.025 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44218.log.gz 2024-04-25T11:47:22.025 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44512.log.gz 2024-04-25T11:47:22.026 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34294.log.gz 2024-04-25T11:47:22.026 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34398.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34439.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33374.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33158.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33268.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33322.log.gz 2024-04-25T11:47:22.028 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33186.log.gz 2024-04-25T11:47:22.029 INFO:teuthology.orchestra.run.smithi136.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33214.log.gz 88.8% 2024-04-25T11:47:22.029 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33400.log.gz 2024-04-25T11:47:22.029 INFO:teuthology.orchestra.run.smithi136.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33106.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 88.9% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33478.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34216.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33426.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34268.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33348.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33452.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33132.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33242.log.gz 2024-04-25T11:47:22.030 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33296.log.gz 2024-04-25T11:47:22.033 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33844.log.gz 2024-04-25T11:47:22.036 INFO:teuthology.orchestra.run.smithi136.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-04-25T11:47:22.036 INFO:teuthology.orchestra.run.smithi136.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34647.log.gz 2024-04-25T11:47:22.036 INFO:teuthology.orchestra.run.smithi136.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34640.log.gz 2024-04-25T11:47:22.037 INFO:teuthology.orchestra.run.smithi136.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-04-25T11:47:22.037 INFO:teuthology.orchestra.run.smithi136.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34658.log.gz 2024-04-25T11:47:22.037 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34659.log.gz 2024-04-25T11:47:22.044 INFO:teuthology.orchestra.run.smithi136.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.33708.log.gz 2024-04-25T11:47:22.053 INFO:teuthology.orchestra.run.smithi136.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.27561.log.gz 2024-04-25T11:47:22.062 INFO:teuthology.orchestra.run.smithi136.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39515.log.gz 2024-04-25T11:47:22.107 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T11:47:22.197 INFO:teuthology.orchestra.run.smithi136.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T11:47:22.801 INFO:teuthology.orchestra.run.smithi136.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T11:47:23.311 INFO:teuthology.orchestra.run.smithi157.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27756.log.gz 2024-04-25T11:47:23.542 INFO:teuthology.orchestra.run.smithi136.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-04-25T11:47:23.589 INFO:teuthology.orchestra.run.smithi157.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T11:47:23.697 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T11:47:30.240 INFO:teuthology.orchestra.run.smithi157.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T11:47:30.712 INFO:teuthology.orchestra.run.smithi157.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T11:47:33.198 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T11:47:33.861 INFO:teuthology.orchestra.run.smithi157.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T11:47:35.441 INFO:teuthology.orchestra.run.smithi157.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T11:47:36.622 INFO:teuthology.orchestra.run.smithi157.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T11:47:51.257 INFO:teuthology.orchestra.run.smithi136.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T11:47:52.777 INFO:teuthology.orchestra.run.smithi157.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-04-25T11:55:59.835 INFO:teuthology.orchestra.run.smithi157.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T11:56:33.157 INFO:teuthology.orchestra.run.smithi157.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T11:58:05.139 INFO:teuthology.orchestra.run.smithi157.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T11:58:27.040 INFO:teuthology.orchestra.run.smithi157.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T11:58:27.042 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-04-25T11:58:27.042 INFO:teuthology.orchestra.run.smithi157.stderr:real 11m5.260s 2024-04-25T11:58:27.042 INFO:teuthology.orchestra.run.smithi157.stderr:user 13m57.212s 2024-04-25T11:58:27.042 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m35.550s 2024-04-25T12:03:15.697 INFO:teuthology.orchestra.run.smithi136.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T12:05:34.013 INFO:teuthology.orchestra.run.smithi136.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T12:07:49.832 INFO:teuthology.orchestra.run.smithi136.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T12:08:34.688 INFO:teuthology.orchestra.run.smithi136.stderr: 96.6% -- replaced with /var/log/ceph/ceph-client.0.35981.log.gz 2024-04-25T12:08:34.690 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-25T12:08:34.690 INFO:teuthology.orchestra.run.smithi136.stderr:real 21m12.910s 2024-04-25T12:08:34.690 INFO:teuthology.orchestra.run.smithi136.stderr:user 19m13.109s 2024-04-25T12:08:34.691 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m54.720s 2024-04-25T12:08:34.691 INFO:tasks.ceph:Archiving logs... 2024-04-25T12:08:34.692 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi136/log 2024-04-25T12:08:34.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T12:09:46.964 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi157/log 2024-04-25T12:09:46.965 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T12:10:38.528 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T12:10:38.538 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T12:10:38.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T12:10:38.541 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T12:10:38.641 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T12:10:38.651 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T12:10:38.652 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T12:10:38.659 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout:+hv01.front.sepi 67.205.162.81 3 u 38 64 377 0.085 -1.942 0.788 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout:+hv02.front.sepi 63.231.80.2 3 u 39 64 377 0.085 -0.178 0.522 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout:*hv03.front.sepi 74.6.168.72 3 u 43 64 377 0.111 -0.824 0.604 2024-04-25T12:10:38.698 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T12:10:38.729 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T12:10:38.729 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-04-25T12:10:38.730 INFO:teuthology.orchestra.run.smithi157.stdout:+hv01.front.sepi 67.205.162.81 3 u 47 128 377 0.096 -1.199 1.206 2024-04-25T12:10:38.730 INFO:teuthology.orchestra.run.smithi157.stdout:+hv02.front.sepi 63.231.80.2 3 u 37 128 377 0.100 +0.155 1.138 2024-04-25T12:10:38.730 INFO:teuthology.orchestra.run.smithi157.stdout:*hv03.front.sepi 74.6.168.72 3 u 48 128 377 0.094 -0.353 1.174 2024-04-25T12:10:38.730 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T12:10:38.731 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T12:10:38.741 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T12:10:38.742 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T12:10:38.751 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T12:10:38.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T12:10:38.770 INFO:teuthology.task.internal:Duration was 3682.752559 seconds 2024-04-25T12:10:38.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T12:10:38.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T12:10:38.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T12:10:38.783 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T12:10:39.020 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T12:10:39.020 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-04-25T12:10:39.021 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T12:10:39.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-04-25T12:10:39.049 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T12:10:39.072 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T12:10:39.072 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T12:10:39.097 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T12:10:39.162 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T12:10:39.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T12:10:39.164 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T12:10:39.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T12:10:39.352 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T12:10:39.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T12:10:39.400 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T12:10:39.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T12:10:39.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T12:10:39.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T12:10:39.473 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-04-25T12:10:39.490 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-04-25T12:10:39.530 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T12:10:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T12:10:39.535 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T12:10:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T12:10:39.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T12:10:39.591 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T12:10:39.592 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi136 2024-04-25T12:10:39.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T12:10:39.618 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673123/remote/smithi157 2024-04-25T12:10:39.618 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T12:10:39.645 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T12:10:39.645 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T12:10:39.657 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T12:10:39.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T12:10:39.700 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T12:10:39.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T12:10:39.710 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T12:10:39.711 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T12:10:39.713 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T12:10:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 12:10 /home/ubuntu/cephtest 2024-04-25T12:10:39.736 INFO:teuthology.orchestra.run.smithi157.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 12:10 /home/ubuntu/cephtest 2024-04-25T12:10:39.737 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T12:10:39.749 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T12:10:39.775 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} duration: 3682.752559185028 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T12:10:39.775 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T12:10:39.854 INFO:teuthology.run:pass